Diffusion approaches to speech enhancement gain immense attention due to their improved speech component reconstruction capability. However, they usually suffer from high inference computational complexity due to multiple iterations during the reverse process. In this paper, we propose EffDiffSE, an efficient diffusion-based frequency-domain speech enhancement model with a hybrid discriminative condition DNN and generative score DNN. Our contributions are three-fold. First, we formulate the powerful time-domain Universe++ model in the frequency domain with a combined psychoacoustic loss and score matching loss. Second, we achieve a single-step efficient reverse process both during training and inference with noise-consistent Langevin dynamics. Third, an auxiliary loss is applied to the single-step reverse process output to improve the diffusion performance further. Trained and evaluated on the URGENT 2024 Speech Enhancement Challenge data splits, the proposed EffDiffSE achieves an MOS comparable to the top reported time- and frequency-domain diffusion baseline methods, while excelling them by >0.2 PESQ points, showing significantly less hallucination and inference computational complexity (3.9 GMAC/s vs. 58 ... 7600 GMAC/s).

Deep neural networks (DNNs) have proven themselves as an effective means to separate clean speech from noisy mixtures. When there are multiple concurrent talkers, however, unambiguously defining the target output is not trivial, especially if the mixture is single-channel and the talkers are not known in advance. Although this problem can be addressed with permutation invariant training or deep clustering, the performance still suffers in this case. Approaches for compact arrays of multiple microphones can exploit spatial diversity to resolve the ambiguity: a separate output may be generated for each direction of arrival (DOA), or the speaker assignment can be controlled with a location-based training (LBT). Alternatively, we can narrow down the target definition at the input, to perform a spatially selective speaker separation instead of separating all speakers simultaneously. This is achieved by specifying freely adjustable target DOAs. On the one hand, these can be integrated as location-based input features (LBI). On the other hand, the main contribution of this work is a location dependent feature extraction (LDE): we implicitly introduce a DOA dependence in a small part of the DNN by optimizing its parameters for each DOA separately. Experiments demonstrate that LDE outperforms LBT and LBI in terms of instrumental metrics and speech recognition results. A representative audio example is presented for a qualitative impression. An analysis of the spatial selectivity reveals that target and nontarget directions can be distinguished quite well with LDE, which is also verified by recordings of real moving talkers.

Recently, BigVGAN has emerged as high-performance speech vocoder. Its sequence-to-sequence-based synthesis, however, prohibits usage in low-latency conversational applications. Our work addresses this shortcoming in three steps. First, we introduce low latency into BigVGAN via implementing causal convolutions, yielding decreased performance. Second, to regain performance, we propose a teacher-student transfer learning scheme to distill the high-delay non-causal BigVGAN into our low-latency causal vocoder. Third, taking advantage of a self-supervised learning (SSL) model, in our case wav2vec2.0, we align its encoder speech representations extracted from our low-latency causal vocoder to the ground truth ones. In speaker-independent settings, both proposed training schemes notably elevate the performance of our low-latency vocoder, closing up to the original high-delay BigVGAN. At only 21% higher complexity, our best small causal vocoder achieves 3.96 PESQ and 1.25 MCD, excelling even the original small non-causal BigVGAN (3.64 PESQ) by 0.32 PESQ and 0.1 MCD points, respectively.

Given a mixture of multiple speech signals, a neural network can extract the talkers individually and sequentially, which may improve the output quality compared to a simultaneous separation of all speakers. To still suppress interfering speech effectively, the residual mixture of the remaining unseparated speakers can be included in the input of the next step. We build upon this approach with a twofold contribution. First, we propose to refine the already extracted speech signals in further optional iterations. This is accomplished by exploiting that the outputs of previous steps provide prior information on interference and target. Experiments indicate a gradual improvement until convergence after about 2 iterations per speaker. Secondly, look directions are defined to control in which order speakers are extracted, thereby resolving the related permutation ambiguity. Whereas supplying the true speaker locations delivers the best results, even a weak guidance with random directions reduces interference leakage significantly.

Speakers are often separated by time-frequency masking in the short-time Fourier domain to take advantage of the high degree of sparsity of the individual speech spectrograms. Magnitude and phase can be jointly enhanced with complex masks, but prior work suggests that directly mapping the input to the complex spectrogram of the clean signal is a better alternative. For a setup with a compact microphone array, experiments conducted in this paper compare these paradigms with focus on magnitude and phase estimation. Whereas phase is enhanced effectively in general, differences between masking and mapping are minor in this regard. Spectral mapping causes the least target distortion. Complex masking better suppresses interference, but speech quality suffers due to artifacts. Combining magnitude masking with phase mapping presents a compromise, which amounts to the best performance regarding instrumental metrics.

Generative speech enhancement methods commonly employ components of text-to-speech (TTS) systems to suppress noise and enhance speech quality. They have won traction recently, as they allow for a clean, virtually noisefree speech estimate. However, they come with unique error types such as mumbled speech and substituted phonemes, which are often not recognized by common nonintrusive speech quality metrics such as NISQA and DNSMOS. Intrusive metrics, such as PESQ and STOI on the other hand, are also not reliable due to their dependence on audio similarity and therefore rarely adopted in TTS research. In this work, we provide insights into typical issues of instrumental evaluation of generative approaches to speech enhancement. Furthermore, we propose the Levenshtein phoneme distance (LPD) that helps to catch and interpret the unique error types evoked by generative approaches. Finally, we propose best practices for interpreting metrics for generative approaches, pointing out that PESQ is indeed useful for the evaluation of generative speech enhancement in low-SNR conditions, while NISQA and DNSMOS are good in mid to high SNR.

Generative approaches to speech enhancement using a vocoder to synthesize a clean speech estimate aim at solving the problem of residual noise occuring with typical maskbased spectral estimation approaches. The necessity to restrict the system’s knowledge to only clean speech and to prevent the possibility of noise reconstruction has recently motivated the introduction of a sparse autoencoder (AE) bottleneck using a pre-trained vector quantizer codebook. In our work, inspired from information bottleneck theory, we propose a maximum entropy information bottleneck (MEIB) regularization, which we derive for the deterministic AE with quantized bottleneck on time series. Furthermore, we introduce a feature-matching regularization encouraging noisy inputs and clean inputs to select the same vector quantizer symbols. The proposed methods significantly elevate our denoising performance by 0.23 PESQ points and 0.06 DNSMOS points.

Fully convolutional recurrent neural networks (FCRNs) have shown state-of-the-art performance in single-channel speech enhancement. However, the number of parameters and the FLOPs/second of the original FCRN are restrictively high. A further important class of efficient networks is the CRUSE topology, serving as reference in our work. By applying a number of topological changes at once, we propose both an efficient FCRN (FCRN15), and a new family of efficient convolutional recurrent neural networks (EffCRN23, EffCRN23lite). We show that our FCRN15 (875K parameters) and EffCRN23lite (396K) outperform the already efficient CRUSE5 (85M) and CRUSE4 (7.2M) networks, respectively, w.r.t. PESQ, DNSMOS and DeltaSNR, while requiring about 94% less parameters and about 20% less #FLOPs/frame. Thereby, according to these metrics, the FCRN/EffCRN class of networks provides new best-in-class network topologies for speech enhancement.

With a microphone array, spatial diversity can be exploited to estimate time-frequency masks that effectively suppress interfering speakers as well as noise. Here, we propose a deep learning approach where the signal components are distinguished based on the associated directions of arrival. To capture the target signal spectrogram more accurately, the estimation can be performed for each subband separately. In order to also take advantage of cross-band dependencies, we additionally consider a combined subband and full-band architecture. Our evaluation indicates that this combination consistently improves the performance in terms of instrumental quality metrics as compared to a pure subband or full-band method. Further, the comparison with two baseline approaches demonstrates the effectiveness of the location based deep learning approach.

Supervised learning methods are a powerful tool for direction of arrival (DOA) estimation because they can cope with adverse conditions where simplified models fail. In this work, we consider a previously proposed convolutional neural network (CNN) approach that estimates the DOAs for multiple sources from the phase spectra of the microphones. For speech, specifically, the approach was shown to work well even when trained entirely on synthetically generated data. However, as each frame is processed separately, temporal context cannot be taken into account. This prevents the exploitation of interframe signal correlations, and the fact that DOAs do not change arbitrarily over time. We therefore consider two different extensions of the CNN: the integration of a long short-term memory (LSTM) layer, or of a temporal convolutional network (TCN). In order to accommodate the incorporation of temporal context, the training data generation framework needs to be adjusted. To obtain an easily parameterizable model, we propose to employ Markov chains to realize a gradual evolution of the source activity at different times, frequencies, and directions, throughout a training sequence. A thorough evaluation demonstrates that the proposed configuration for generating training data is suitable for the tasks of single-, and multi-talker localization. In particular, we note that with temporal context, it is important to use speech, or realistic signals in general, for the sources. Experiments with recorded impulse responses and noise reveal that the CNN with the LSTM extension outperforms all other considered approaches, including the plain CNN, and the TCN extension.

The Interspeech 2020 Deep Noise Suppression (DNS) Challenge focuses on evaluating low-latency single-channel speech enhancement algorithms under realistic test conditions. Our contribution to the challenge is a method for joint dereverberation and denoising based on complex spectral mask estimation using a fully convolutional recurrent network (FCRN) which relies on a convolutional LSTM layer for temporal modeling. Since the effects of reverberation and noise on perceived speech quality can differ notably, a multi-target loss for controlling the weight on desired dereverberation and denoising is proposed. In the crowdsourced subjective P.808 listening test conducted by the DNS Challenge organizers, the proposed method shows a significant overall improvement of 0.43 MOS points over the DNS Challenge baseline and ranks amongst the top-3 submissions for both realtime and non-realtime tracks of the challenge.

Single-channel speech enhancement in highly non-stationary noise conditions is a very challenging task, especially when interfering speech is included in the noise. Deep learning-based approaches have notably improved the performance of speech enhancement algorithms under such conditions, but still introduce speech distortions if strong noise suppression shall be achieved. We propose to address this problem by using a two-stage approach, first performing noise suppression and subsequently restoring natural sounding speech, using specifically chosen neural network topologies and loss functions for each task. A mask-based long short-term memory (LSTM) network is employed for noise suppression and speech restoration is performed via spectral mapping with a convolutional encoder-decoder network (CED). The proposed method improves speech quality (PESQ) over state-of-the-art single-stage methods by about 0.1 points for unseen highly non-stationary noise types including interfering speech. Furthermore, it is able to increase intelligibility in low-SNR conditions and consistently outperforms all reference methods.

Convolutional recurrent neural networks (CRNs) using convolutional encoder-decoder (CED) structures have shown promising performance for single-channel speech enhancement. These CRNs handle temporal modeling through integrating long short-term memory (LSTM) layers in between convolutional encoder and decoder. However, in such a CRN, the organization of internal representations in feature maps and the focus on local structure of the convolutional mappings has to be discarded for fully-connected LSTM processing. Furthermore, CRNs can be quite restricted concerning the feature space dimension at the input of the LSTM, which, through its fully-connected nature, requires a large amount of trainable parameters. As first novelty, we propose to replace the fully-connected LSTM by a convolutional LSTM (ConvLSTM) and call the resulting network a fully convolutional recurrent network (FCRN). Secondly, since the ConvLSTM retains the structured organization of its input feature maps, we can show that this helps to internally represent the harmonic structure of speech, allowing us to handle high-dimensional input features using less trainable parameters than an LSTM. The proposed FCRN clearly outperforms CRN reference models with similar amounts of trainable parameters in terms of PESQ, STOI, and segmental ΔSNR.

The weighted least-squares (WLS) direction-of-arrival estimator that minimizes an error based on interchannel phase differences is both computationally simple and flexible. However, the approach has several limitations, including an inability to cope with spatial aliasing and a sensitivity to phase wrapping. The recently proposed phase wrapping robust (PWR)-WLS estimator addresses the latter of these issues, but requires solving a nonconvex optimization problem. In this contribution, we focus on both of the described shortcomings. First, a conceptually simpler alternative to PWR is presented that performs comparably given a good initial estimate. This newly proposed method relies on an unwrapping of the phase differences vector. Secondly, it is demonstrated that all microphone pairs can be utilized at all frequencies with both estimators. When incorporating information from other frequency bins, this permits a localization above the spatial aliasing frequency of the array. Experimental results show that a considerable performance improvement is possible, particularly for arrays with a large microphone spacing.

For blind speech dereverberation, two frameworks are commonly used: on the one hand, the multi-channel linear prediction (MCLP) framework, and on the other hand, data-dependent beamforming, e.g., the generalized sidelobe canceler (GSC) framework. The MCLP framework is designed to perform deconvolution and hence has gained increased prominence in blind speech dereverberation. The GSC framework is commonly used for noise reduction, but may be applied for dereverberation as well. In previous work, we have shown that for the noiseless case, MCLP and the GSC yield in theory mathematically equivalent results in terms of dereverberation. In this paper, we assume additional coherent as well as incoherent-noise components and formally analyze and compare both frameworks in terms of dereverberation and noise reduction performance. Both the theoretical analysis and time domain simulation results demonstrate that unlike the GSC, MCLP expectably shows limited performance in terms of noise reduction, while both perform equally well in terms of dereverberation, provided that the GSC blocking matrix achieves complete blocking of the early reverberant-speech component and sufficiently many microphones are available. In case of incomplete blocking, however, the GSC performs inferior to MCLP in terms of dereverberation, as shown in short-time Fourier transform domain simulations.

Regression based on neural networks (NNs) has led to considerable advances in speech enhancement under non-stationary noise conditions. Nonetheless, speech distortions can be introduced when employing NNs trained to provide strong noise suppression. We propose to address this problem by first suppressing noise and subsequently restoring speech with specifically chosen NN topologies for each of these distinct tasks. A mask-estimating long short-term memory (LSTM) network is employed for noise suppression, while the speech restoration is performed by a fully convolutional encoder-decoder (CED) network, where we introduce temporal modeling capabilities by using a convolutional LSTM layer in the bottleneck. We show considerable performance gains over reference methods of up to 0.26 MOS points (PESQ) and the ability to significantly improve intelligibility in terms of STOI for low-SNR conditions.

In this contribution we present our latest investigations and analysis on a novel a priori SNR estimator for speech enhancement applications. It is based on a clean spectral envelope estimation with a deep neural network (DNN) in the cepstral domain. Furthermore, by integrating our cepstral excitation manipulation (CEM) approach into this framework, we obtain not only a smooth and natural background noise experience, but also achieve noise reduction between harmonics which is not possible with low-order models. We investigate the performance of the proposed approach in conjunction with three different spectral weighting rules and show improvement of more than 3.5 dB noise attenuation vs. the well-known decision-directed (DD) approach without a significant trade-off in speech distortion.

In this paper, we propose and compare various techniques for the estimation of clean spectral envelopes in noisy conditions. The source-filter model of human speech production is employed in combination with a hidden Markov model and/or a deep neural network approach to estimate clean envelope-representing coefficients in the cepstral domain. The cepstral estimators for speech spectral envelope-based noise reduction are both evaluated alone and also in combination with the recently introduced cepstral excitation manipulation (CEM) technique for a priori SNR estimation in a noise reduction framework. Relative to the classical MMSE short time spectral amplitude estimator, we obtain more than 2 dB higher noise attenuation, and relative to our recent CEM technique still 0.5 dB more, in both cases maintaining the quality of the speech component and obtaining considerable SNR improvement.

Various studies have shown that the instrumental measures wideband PESQ and POLQA are not reliably predicting speech quality for artificial speech bandwidth extension (ABE) test conditions, as this has never been their scope. Based on data from a coordinated subjective listening test with 12 ABE variants developed by 6 different institutions, conducted in 4 languages, we propose in this work a novel instrumental quality measure that is specifically suited for narrowband-to-wideband ABE test conditions. In particular, our contributions are fourfold: First, we propose quality indicators particularly being able to detect ABE-related distortions. Second, we investigate the combination of perceptually and nonperceptually motivated distortion-related statistics. Third, we propose a support-vector-machine-based high-performance MOS predictor for ABE speech quality assessment, finally, we present the training process based on the subjective listening test data. A k-fold cross-validation test on 1) disjoint languages, 2) disjoint speakers, and 3) disjoint ABE solutions proves the superiority of our proposed measure in the ITU-T-recommended categories accuracy, consistency, and linearity compared to both, wideband PESQ and POLQA.

The development of new speech enhancement techniques is a continuous progress to combat the impairment of speech signals by various acoustical environmental influences. In this contribution we propose a new two-stage speech enhancement algorithm, exploiting the source-filter model to decompose a denoised target signal, and specifically we manipulate the excitation signal in the cepstral domain. The second stage therein is a refinement of the a priori signal-to-noise ratio (SNR) estimate used for the suppression gain calculation. Different to prior art, a higher noise attenuation can be achieved, without any more artifacts in the processed speech component.