We propose a novel training framework for lightweight speaker verification (SV) systems, which are often deployed on mobile devices with constrained storage resources. Our method leverages supervised contrastive loss to learn speaker-discriminative embeddings. To enhance negative sample selection, crucial in contrastive learning, we introduce CurriNeg, a curriculum learning strategy that selects negative pairs based on their difficulty. A teacher network assigns difficulty scores to negative pairs, guiding a student network to learn progressively from easy to hard negative pairs. Additionally, we propose to leverage AM-Softmax’s capability to form multi-class boundaries to compensate for the absence of multi-class optimization in the contrastive loss function, which results in the CurriNeg-AMS loss. Experiments on VoxCeleb1 show that our method reduces relative equal error rate by 8.6% and 20.2% over the X-vector and Fast ResNet34 baselines, respectively. Code is available at: GitHub.1

Automatic speaker verification and audio anti-spoofing perform unreliably under domain mismatch conditions, such as a mismatch in channels and recording devices. Domain generalization aims to ensure reliable performance on unseen conditions while maintaining high discrimination on seen domains. This paper proposes a novel method UDANet (Uncertainty-guided Domain Augmentation Network) for domain generalization. The key idea of UDANet is to augment the source domain in neural representation through a Uncertainty-guided Mock Domain Generator (UMDG). The UMDG uses a Gaussian mixture density network to learn class features that are robust across domains. This uncertainty guides domain augmentation toward ambiguous representations, generating more diverse and challenging domain samples. By augmenting uncertain neural representations, this approach expands the model’s distributional space and enhances robustness across different domains. Theoretical analyses prove that UMDG matches target shift moments to upper-bound target risk, guaranteeing a more domain robust model. Experimental results demonstrate that the UDANet improves performance on standard benchmarks, especially in cross-database and challenging spoofing scenarios. Code is available at GitHub.1

Speaker verification and anti-spoofing systems often face significant performance degradation when encountering out of-domain (OOD) scenarios due to the variability in speaker characteristics, recording environments, and unknown spoofing attacks. Domain generalization (DG) methods are designed to identify features that remain consistent across various domains, thereby enhancing the robustness of deep learning models. Style augmentation is a DG approach that synthesizes novel domain features from feature statistics. However, previous works only consider the covariances within individual mini-batches during style augmentation, which may lead to outliers in the augmented samples. In this paper, we propose to improve the generalization capability of speaker verification and anti-spoofing systems by leveraging the posterior covariances of neural representations in the outputs of Transformer layers of a neural network through Bayesian adaptation and call the method Bayesian Neural Representation Augmentation (BNRA). We use the covariances estimated from the current mini-batch to define the covariances of a Gaussian likelihood function for the observed data. The covariances of the prior distribution of latent factors are updated recursively. Our plug-and-play module can be seamlessly integrated into existing self-supervised learning networks without extra parameters. It is the first to incorporate augmentation of neural representations for OOD generalization in speaker verification and anti-spoofing, with extensive experiments confirming its consistent performance gains.

The performance of automatic speaker verification (ASV) and anti-spoofing drops seriously under real-world domain mismatch conditions. The relaxed instance frequency-wise normalization (RFN), which normalizes the frequency components based on the feature statistics along the time and channel axes, is a promising approach to reducing the domain dependence in the feature maps of a speaker embedding network. We advocate that the different frequencies should receive different weights and that the weights' uncertainty due to domain shift should be accounted for. To these ends, we propose leveraging variational inference to model the posterior distribution of the weights, which results in Bayesian weighted RFN (BWRFN). This approach overcomes the limitations of fixed-weight RFN, making it more effective under domain mismatch conditions. Extensive experiments on cross-dataset ASV, cross-TTS anti-spoofing, and spoofing-robust ASV show that BWRFN is significantly better than WRFN and RFN.

His paper presents a method for enhancing multi-speaker audio generation using limited samples based on acoustic signals. The proposed approach improves the existing variational inference text-to-speech (VITS) model by incorporating the vector quantized variational autoencoder (VQVAE) network and the attention mechanism to enhance audio generation quality for different speakers. Traditional VITS models trained on large and high-quality datasets can generate high-quality text-to-speech outputs with good generalization. However, with small datasets, the VITS model often struggles to maintain audio quality and speaker characteristics. To address these limitations, we introduce the VQVAE network to discretize continuous acoustic features into tokens, effectively capturing essential speaker characteristics such as timbre and articulation. The attention mechanism further improves the model's ability to focus on these critical speaker-specific features, especially in multi-speaker scenarios. Experimental results show that our enhanced model not only excels in generating natural and high-quality speech for the original speaker but also effectively generalizes to other speakers. This method demonstrates significant improvements in naturalness and speaker characteristic retention, making it particularly effective for multi-speaker audio generation in scenarios with limited data.

Existing traditional ocean vertical-mixing schemes are empirically developed without a thorough understanding of the physical processes involved, resulting in a discrepancy between the parameterization and forecast results. The uncertainty in ocean-mixing parameterization is primarily responsible for the bias in ocean models. Benefiting from deep-learning technology, we design the Adaptive Fully Connected Module with an Inception module as the baseline to minimize bias. It adaptively extracts the best features through fully connected layers with different widths, and better learns the nonlinear relationship between input variables and parameterization fields. Moreover, to obtain more accurate results, we impose KPP (K-Profile Parameterization) and PP (Pacanowski–Philander) schemes as physical constraints to make the network parameterization process follow the basic physical laws more closely. Since model data are calculated with human experience, lacking some unknown physical processes, which may differ from the actual data, we use a decade-long time record of hydrological and turbulence observations in the tropical Pacific Ocean as training data. Combining physical constraints and a nonlinear activation function, our method catches its nonlinear change and better adapts to the ocean-mixing parameterization process. The use of physical constraints can improve the final results.

Image dehazing has always been a challenging visual task. Most of the existing image dehazing algorithms are constructed based on convolutional neural networks, which perform very well in capturing local features and detail information, but fall short of the ability to capture remote contextual information. In this paper, we introduce a novel method called CAAVM-TransUNet. The method integrates Transformer into the traditional U -shaped architecture. Unlike earlier Transformer-based U-Net, CAA VM-TransUNet introduces the State Space Sequence Model (SSM) to enhance the ability of capturing remote dependencies, and adds Context Anchor Attention (CAA) to filter redundant features in the hopping connectivity layer prior to the encoder and decoder, so that the model can better focus on the key feature information and reduce the interference of irrelevant information, thus improving the effect of image dehazing. The results show that the dehazing effect of CAAVM-TransUNet on RESIDE dataset is better than the existing methods, especially the PSNR in the SOTS-outdoor dataset reaches 36.13 dB, which provides a new idea to solve the complex image dehazing problem.

In recent years, using raw waveforms as input to deep networks has been widely explored for speaker verification systems that process speech signals at the segmental level. A critical issue of such an approach is that the front-end network with a small kernel fails to capture the suprasegmental information, such as the intonation patterns and prosody that span longer than one second. This paper proposes a novel framework that can capture the segmental and suprasegmental information after the first convolutional layer. Concretely, suprasegmental information is obtained from the first-order finite difference of two consecutive suprasegmental envelopes estimated by Hilbert transforms. Experimental evaluations on the Voxceleb dataset show that combining segmental and suprasegmental features can reduce the EER of an end-to-end system by 27%. To our best knowledge, this is the first attempt to incorporate suprasegmental information for end-to-end speaker verification.

Transformer based end-to-end modelling approaches with multiple stream inputs have been achieved great success in various automatic speech recognition (ASR) tasks. An important issue associated with such approaches is that the intermediate features derived from each stream might have similar representations and thus it is lacking of feature diversity, such as the descriptions related to speaker characteristics. To address this issue, this paper proposed a novel multi-level acoustic feature extraction framework that can be easily combined with Transformer based ASR models. The framework consists of two input streams: a shallow stream with high-resolution spectrograms and a deep stream with low-resolution spectrograms. The shallow stream is used to acquire traditional shallow features that is beneficial for the classification of phones or words while the deep stream is used to obtain utterance-level speaker-invariant deep features for improving the feature diversity. A feature correlation based fusion strategy is used to aggregate both features across the frequency and time domains and then fed into the Transformer encoder-decoder module. By using the proposed multi-level acoustic feature extraction framework, state-of-the-art word error rate of 21.7% and 2.5% were obtained on the HKUST Mandarin telephone and Librispeech speech recognition tasks respectively.

Flocculation and sedimentation is a crucial step in the water treatment process. Currently, most water plants still use a fixed-value proportional dosing method for flocculant dosing, which has low accuracy. Flocculant dosing prediction is a time series problem, and the complexity of the problem that can be expressed using traditional time-series modeling is limited, and machine learning requires a more complex manual feature engineering component. In this paper, we propose an LSTM neural network prediction model incorporating the Attention mechanism to correlate current sensor acquisition data with historical moment data, extract multidimensional features, and focus on key information and ignore redundant information. It can be a better solution for this problem with nonlinearity, multiple input factors, uncertainty, and time-varying characteristics. Through experiments, comparing the common models such as BP, RNN, LSTM, etc. to predict the flocculant dosing of half-yearly in water plants, the model has a high accuracy.

In recent years, using raw waveforms as input for deep networks has been widely explored for the speaker verification system. For example, RawNet and RawNet2 extracted speaker's feature embeddings from waveforms automatically for recognizing their voice, which can vastly reduce the front-end computation and obtain state-of-the-art performance. However, these models do not consider the speaker's high-level behavioral features, such as intonation, indicating each speaker's universal style, rhythm, etc. This paper presents a novel network that can handle the intonation information by computing the finite difference of different speakers' utterance variations. Furthermore, a hierarchical way is also designed to enhance the intonation property from coarse to fine to improve the system accuracy. The high-level intonation features are then fused with the low-level embedding features. Experimental results on official VoxCeleb1 test data, VoxCeleb1-E, and VoxCeleb-H protocols show our method outperforms and robustness existing state-of-the-art systems. To facilitate further research, code is available at https://github.com/happyjin/FDN

As people's pursuit of large screen-to-body ratio screen experience continues to improve, neither the digging front camera nor the bangs front camera can meet people's requirements for the front camera of a mobile phone. Therefore, the research and development of full-screen equipment has become a new trend. A full-screen device requires the imaging device to be placed below the screen, which we call an under-display cameras. The under-display cameras will improve the user's interactive experience while expanding the screen-to-body ratio of the mobile phone. However, there are many problems in the development of under-display cameras. When the imaging device is installed under the screen, the lower light transmittance will cause serious image degradation. Therefore, a new U-Net, which we call residual dense attention UNet (RDAU-Net), is proposed in this paper. A residual dense attention module which we propose in RDAU-Net to replace the single-layer convolution in the U-Net network. Meanwhile, the introduction of channel attention can effectively enhance the interdependence between channels, thereby adaptively re-dividing channel features. Experiments show that our RDAU-Net has better accuracy and faster recovery efficiency than existing methods.

Symbolic melodies generation is one of the essential tasks for automatic music generation. Recently, models based on neural networks have had a significant influence on generating symbolic melodies. However, the musical context structure is complicated to capture through deep neural networks. Although long short-term memory (LSTM) is attempted to solve this problem through learning order dependence in the musical sequence, it is not capable of capturing musical context with only one note as input for each time step of LSTM. In this paper, we propose a novel Enhanced Memory Network (EMN) with several recurrent units, named Enhanced Memory Unit (EMU), to explicitly modify the internal architecture of LSTM for containing music beat information and reinforces the memory of the latest musical beat through aggregating beat inside the memory gate. In addition, to increase the diversity of generated musical notes, cosine distance among adjacent time steps of hidden states is considered as part of loss functions to avoid a high similarity score that harms the diversity of generated notes. Objective and subjective evaluation results show that the proposed method achieves state-of-the-art performance. Code and music demo are available at https://github.com/qrqrqrqr/EMU

The Transformer has shown tremendous progress in Automatic Speech Recognition (ASR), outperforming recurrent neural network-based approaches. Transformer architecture is good at parallelizing data to accelerate as well as capturing content-based global interaction. However, most studies with Transformer have been utilized only shallow features extracted from the backbone without taking advantage of the deep feature that possesses invariant property. In this paper, we propose a novel framework with the Two Streams and Two Resolution spectrograms Model (TSTRM) that consists of different resolution spectrograms for different stream aiming to capture both shallow and deep features. The feature extraction module consists of a deep network for low-resolution spectrogram and a shallow network for high-resolution spectrogram. The backbone obtains not only detailed acoustic information for speech-text alignment but also utterancelevel representation that contains speaker information. Both features are fused with our proposed fusion method and then input into the Transformer encoder-decoder. The proposed framework shows the state-of-the-art results on the HKUST Mandarin telephone and Librispeech corpora. To the best of our knowledge, this is the first investigation of incorporating deep features to the backbone and use both low and high resolutions spectrogram to focus on global and local information. Code is available at https://github.com/happyjin/TSTRM