In recent years, the emergence of disruptive deepfake technology (referring here to computer-generated speech/audio, images, video and text) has raised significant concerns surrounding the privacy, security, and credibility of digital content. Although detection systems report high benchmark performance, most are evaluated on in-domain datasets, codecs, and generative models seen during training. In real deployment, however, detectors must contend with out-of-domain instances, that is, a mismatch between the data seen during training and the conditions encountered in deployment (e.g., unseen generative methods, unfamiliar codecs, or signals degraded by transmission channels). Under these conditions, performance often degrades sharply.While robustness has been acknowledged in the literature, it remains underexplored. This work presents a robustness-focused survey and experimental study of deepfake speech detection. Rather than broadly cataloguing existing methods, we organise the literature around three factors that influence robustness to distribution shift: dataset design, feature representations, and model architecture. To complement this synthesis, we conduct a cross-dataset evaluation with Wav2Vec2, HuBERT, WavLM, and Whisper self-supervised speech models, trained on the ASVspoof 2019 LA and evaluated on the ASVspoof 2021 Deepfake dataset.The experimental results show that robustness varies considerably across models and conditions. HuBERT demonstrates comparatively stable performance across codec and dataset shifts, while Wav2Vec2 and WavLM experience substantial degradation, particularly under MP3-based compression and neural vocoder synthesis. A diagnostic analysis further reveals that robustness failures are strongly associated with codec-induced masking of synthesis artefacts and with dataset-dependent synthesis pipelines.By combining a robustness-oriented survey with cross-dataset experimental evidence, this work puts theory to practice, highlighting key factors that limit the generalisation of current detectors and outlining practical directions for improving robustness, including diversified dataset design, feature fusion, and evaluation under realistic cross-domain conditions.
In this article, we contrast different subjective tests for the evaluation of text-to-speech systems. We compare results obtained via MOS, a five-point absolute rating test, to two direct comparison tests: the AB test and the Best Worst Scaling (BWS) test. In an AB test, participants select their preferred stimulus from two options, while in a BWS test, they select the best and worst from multiple stimuli. We evaluate these three tests in terms of session duration, participant consistency, number of significant differences detected, and system ranking. Additionally, we estimate the impact of number of participants and test duration in their performance. To make this comparison, we conducted three separate listening tests (MOS, AB and BWS) using a common set of stimuli and analysed results derived from all collected ratings, as well as results derived from reduced subsets of listeners and stimuli. Our findings indicate that BWS tests provide more user ratings within the same amount of time. We also found that participants are more consistent with each other and themselves in the BWS test. Our results also show that the BWS test reveals more significant differences than both AB and MOS, especially with fewer participants or with shorter tests. In particular, we found that a BWS test with as few as 10 participants reveals as many differences as a MOS test with 20.
Sequence labeling is a fundamental while challenging task in natural language processing. In this paper, we propose a language model with a stroke-based scheme, called contextual stroke embedding (CSE), for Chinese sequence labeling. The proposed scheme, which is based on the inner structure of Chinese characters, aims to extract rich internal semantic information from Chinese stroke sequence as well as contextual syntactic information among Chinese characters. The scheme works by first decomposing Chinese character into stroke sequence. The resulting sequences are then used to train a stroke-level language model, which contains a two-layer BiLSTM network architecture to produce character representations. The resulting method has been evaluated on various Chinese sequence labeling tasks, including named entity recognition, Chinese word segmentation and part-of-speech, and compared with related methods. The results confirm the significance of the devised CSE. Specifically, the CSE is able to capture Chinese semantic structural information, thus greatly improving the performance of proposed method on various Chinese sequence labeling tasks. Further, the results also show that our method could achieve the best performance among non-BERT methods for Chinese sequence labeling and work well with BERT.
Speech separation, a major advancement in audio processing, not only enhances the quality of calls and recordings but also unlocks vast potential in applications such as speech recognition, virtual assistants, and sound restoration in noisy environments. In recent years, deep learning-based voice separation approaches have demonstrated remarkable performance. However, reducing model size while maintaining high performance remains a significant challenge, particularly for resource-constrained scenarios and low-resource applications. In this paper, we introduce an innovative approach called Selective Teaching Yourself (STY), designed to enhance the efficiency of monaural utterance separation. The STY strategy employs a shallower frame to extract essential features from a deeper frame within the same model architecture, thereby improving overall performance through selective knowledge sharing. Compared to independently trained models, STY exhibits superior generalization and effectiveness on the evaluated backbone architectures. Notably, despite having only 2.3M parameters, our STY strategy achieves competitive inference performance compared to large-scale state-of-the-art frameworks.
This paper proposes a solution to the well-known long-term problem in speech enhancement that concerns how to provide gains in signal-to-noise ratio (SNR) without degrading speech recognition accuracy. Our approach consists of generating the artifact and residual noise and subtracting them from the enhanced speech. It does not require training automatic speech recognition (ASR) engines with the speech delivered by enhancement schemes. The proposed architecture, NAPGAN (noise and artifact projection GAN, Generative Adversarial Network), is trained to generate the artifact and residual noise modeled by orthogonal projection of the enhanced speech onto the speech-noise subspace. Both the generated artifact and residual noise are subsequently removed from the enhanced speech. Experiments with three deep-learning-based beamforming or speech enhancement schemes show that NAPGAN can lead to SNR gains between 1 dB and 4 dB without a degradation in ASR accuracy in some cases, and with statistically significant reductions of WER in others. This is consistent with results observed using Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI) metrics that either remain the same or slightly improve after applying NAPGAN.
Recent research has revealed that linguistic features are effective digital markers for the early detection of Alzheimer’s disease (AD). An increasing number of studies has developed automatic methods for feature extraction; however, most of them focused on alphabetical languages and lacked direct and multifaceted comparisons between the linguistic features derived from automatic versus manual transcriptions, particularly in evaluating their effectiveness in early patient screening. To bridge this gap, the present study develops an automatic pipeline for linguistic feature extraction using a self-collected Chinese speech dataset and, critically, we conduct comprehensive comparisons through correlation analysis, machine learning classifiers and feature importance ratings. The correlation results suggested moderate to strong agreement between features extracted from the two approaches. Classifiers trained on manual and automatic transcriptions yielded statistically comparable results. Manual transcriptions demonstrated a slight numerical advantage, achieving a peak accuracy of 0.825 compared to 0.800 for their automated counterparts. Important features contributing to the classifiers also demonstrated considerable overlap across the two transcription approaches, including type-to-token ratio (TTR), long pause ratio (LPR) and number of unique words (UW) as key indicators. Overall, the results demonstrated parallel effectiveness of linguistic features from automatic and manual transcriptions, indicating the clinical feasibility of this automatic feature extraction pipeline for early AD detection in Chinese populations.
Hearing aid users frequently struggle to understand speech in noisy environments, negatively impacting their quality of life. Inspired by recent progress in speech technology through community-driven machine learning challenges, the Clarity project launched the first-ever Clarity Enhancement Challenge (CEC1). This challenge specifically addressed speech-in-noise enhancement for hearing aids, uniquely combining objective and subjective intelligibility evaluations to assess performance. Participants developed algorithms aimed at improving speech intelligibility in a simulated domestic environment featuring a target speaker and a stationary noise interferer—either competing speech or domestic appliances. Competitors were provided with an open-source dataset, comprising a novel 40-speaker British English corpus, realistic domestic noise samples, and a baseline hearing aid model with basic signal processing.This paper describes the design and outcomes of CEC1. Thirteen entries were evaluated objectively using the Modified Binaural Short-Time Objective Intelligibility metric (MBSTOI) and subjectively by a listening panel of hearing-impaired individuals. The majority of systems employed deep neural networks (DNNs), classical beamforming, or a combination of both. Results showed significant intelligibility gains over the baseline, particularly for systems combining adaptive beamforming with neural network-based noise reduction. However, algorithms optimised directly for MBSTOI scores did not always translate to real-world listening benefits, highlighting the critical importance of perceptual evaluation in assessing intelligibility. These findings underscore the potential of machine-learning-driven approaches for enhancing hearing aid performance and set a foundation for future challenges addressing dynamic and more realistic auditory scenarios.
Automatic speech recognition (ASR) systems are increasingly used as transcription tools in psycholinguistic research. The present study examines whether Whisper large-v3 parallels human sensitivity to morphosyntactic variability in speech production. We focus on Spanish grammatical gender agreement in regular and dual-gendered nouns (DGNs). DGNs are feminine nouns that begin with stressed /a/ and often take masculine determiners (el agua), creating ambiguity between prescriptive rules and real-world usage. Using a controlled sentence-repetition paradigm, we compared native Spanish speakers and Whisper large-v3 in repetition/transcription accuracy and correction behavior when processing determiner-noun gender mismatches. Both humans and Whisper corrected prescriptively ungrammatical input for regular nouns. For DGNs, humans exhibited determiner-specific variability in correction shaped by collocational frequency, whereas Whisper defaulted to faithful transcription regardless of grammaticality or collocational frequency. These results indicate that Whisper behaves as though it favors prescriptively dominant forms in grammatically stable contexts, but shows little adjustment in the absence of reliable distributional cues. Confidence-score and n-best analyses suggested that, in rare DGN correction cases, the prescriptively incorrect determiner produced by humans often remained available among alternative hypotheses, but received a lower decoding score than the prescriptively correct form. Although Whisper can serve as an efficient transcription tool, high transcription accuracy does not entail human-like language processing. Structured human verification and linguistically informed evaluation remain essential when Whisper is used to transcribe or analyze morphosyntactic variation. More broadly, structured grammatical variability provides a testbed for improving Whisper evaluation and developing computational models that better align with human spoken-language behavior.
Electroencephalography (EEG)-based speech brain-computer interfaces (BCIs) have gained increasing research interest as a potential means to restore or decode speech for individuals with severe communication impairments, particularly with the recent integration of deep learning techniques. This systematic review, conducted in accordance with the PRISMA 2020 guidelines, provides a comprehensive and quantitative overview of deep learning methods applied to EEG and intracranial EEG (iEEG) speech processing tasks published between 2018 and 2025. A systematic search was conducted across Scopus, IEEE Xplore, ScienceDirect, Web of Science, and PubMed, yielding 1,148 records. Following duplicate removal, screening, and eligibility assessment, 80 peer-reviewed original research articles were included. Studies were organized by task type (speech classification, spectrogram reconstruction, and speech synthesis), neural signal type (non-invasive EEG versus invasive electrocorticography (ECoG) and stereoelectroencephalography (sEEG)), and model architecture to enable structured comparison. Results demonstrate that deep learning models significantly outperform traditional methods in speech classification tasks, while spectrogram reconstruction and speech synthesis remain challenging, particularly for non-invasive EEG. Invasive recordings consistently yield superior performance for reconstruction and synthesis tasks. Despite methodological advances, only a limited number of studies address real-time feasibility or cross-subject generalization, highlighting persistent barriers to clinical translation. Overall, this review provides an integrated, task-oriented synthesis of deep learning-based neural speech decoding and highlights the key methodological and translational barriers that must be addressed to enable robust, real-time, and clinically viable speech neuroprosthetic systems.
This paper investigates the impact of voice similarity on synthetic voice preference and Quality of Experience (QoE), focusing on two key prosodic features: long-term fundamental frequency (LTF0) or characteristic pitch and speech rate (SR). By conducting subjective tests on synthetic voices with varying degrees of similarity to each participant’s own voice, the study aims to uncover whether and how voice similarity influences synthetic voice preference. Thirty-four participants were recorded reading 10 phonetically balanced Harvard sentences. From these data, we extracted LTF0 using CREPE and SR using Wav2Vec-XLSR-53. The same features were extracted from four commercial Google TTS voices. The LTF0 and SR of the TTS voices were then matched with the voices of each participant. In addition to the matched voices, four variations were made for LTF0, while two speed variations were made for SR. Personalised listening tests were prepared and administered to each participant, comparing the matched stimulus with a variation in random order. The results show that the participants preferred the matched stimuli in most cases. In particular, this preference was stronger when voice matching was applied to female TTS voices. For male TTS voices, lower LTF0 was preferred. Moreover, matched and faster SR were preferred over slower SR. These findings suggest that voice adaptation and personalisation can enhance perceived QoE in terms of user preference.
Adapting Speech Foundation Models (SFMs) to intersectional domains remains challenging when multiple distribution shifts such as acoustic conditions, speaking style, or speaker population co-occur and joint supervision data is scarce. Rather than training a separate model for each domain combination, this work studies compositional domain adaptation, asking whether model updates induced by distinct supervision sources can be recombined to generalize to unseen intersections. We investigate this question using child automatic speech recognition (ASR) as a case study, where age-related acoustics and speaking style jointly define a difficult low-resource setting. We propose a structured model merging framework that composes task-specific adaptations without retraining. To enable stable and interpretable composition, we introduce Headwise Selective Attention (HSA) Merge, which restricts parameter arithmetic to salient attention heads where task-specific adaptations are concentrated. We further demonstrate robustness under acoustic mismatch, cross-corpus transfer, dialectal variation, and when source vectors are derived from synthetic or noisy data. Across Whisper model sizes, HSA Merge consistently outperforms single-source fine-tuning by relative WER of up to 20.6% on child ASR and 29.2% on cross-dialect transfer. These results indicate that key factors of variation in ASR models behave as partially separable components and that structured attention-level merging provides a practical mechanism for compositional generalization in low-resource and intersectional ASR settings.
Parkinson’s disease (PD) is a neurodegenerative condition characterized by motor and non-motor symptoms including voice changes in its early stage. The voice deterioration has been used as a biomarker for the detection of PD. Although, the voice analysis shows promising results in specific language, its accuracy reduces when tested in multilingual scenario. In this paper, we propose an instance-based domain adaptation method using kernel mean matching (KMM) to mitigate the distributional shift between different language domains in detecting PD. A hybrid feature extraction technique combining mel frequency cepstral coefficient (MFCC) and time-frequency (TF) features is used as input to the model. The performance of the proposed method is assessed with sustained vowel phonations /a/, /e/, /i/, /o/ and /u/ from different datasets in multiple languages, including Spanish, Italian, and English. The evaluation of the model is performed using leave one out cross validation (LOO-CV) with baseline classifiers: support vector machine (SVM) and K-nearest neighbour (KNN). The experimental results show that the proposed model with KNN provide superior classification accuracy across cross-domain tasks. The highest classification accuracy of 93.87% is achieved for the vowel /o/ in Italian to Spanish adaptation, while 91.42% accuracy for English to Spanish adaptation in vowel /a/ and 84.94% accuracy for Spanish to Italian in vowel /o/. These findings show the potential of the proposed method for improving PD detection performance in diverse datasets in multilingual scenarios.
Speech Emotion Recognition (SER) in low-resource languages is hindered by scarce labeled data, speaker variability and prosodic diversity. Existing few-shot learning models underperform in such settings due to weak generalization and inadequate exploitation of emotional cues in speech. This study proposes SegTM-SGD, a novel meta-learning framework that integrates Whisper-derived acoustic representations with a lightweight Transformer encoder optimized using Meta-SGD. Unlike prior approaches, SegTM-SGD employs segment-level temporal modeling with learnable per-parameter adaptation, enabling rapid task transfer with minimal supervision. The framework adopts episodic N-way K-shot learning to simulate real-world data scarcity. Experiments conducted on Tamil, Malayalam, and Indian-English emotional speech datasets demonstrate significant performance gains over state-of-the-art metric- and gradient-based baselines. SegTM-SGD achieves macro F1-scores of 0.90, 0.83, and 0.92 in 1-shot settings, improving to 0.95, 0.94, and 0.96 under 10-shot conditions, respectively. These results highlight the effectiveness of segment-aware transformer refinement with adaptive meta-learning for few-shot SER in multilingual low-resource environments.
We propose a novel method to enhance dependency parsing by using graph embeddings learned from universal dependency treebanks. Five distinct algorithms are employed to learn embedding vectors that encode diverse information about each node in a treebank graph. These embeddings are then integrated into a head-driven sequence prediction model powered by either a multi-layer bidirectional recurrent neural network or a multi-block transformer-based bidirectional encoder. This model outperforms a baseline model using a state-of-the-art supervised approach on four dependency treebanks, covering both low- and high-resource languages: Vietnamese, Indonesian, French, and English. The new parser, along with all trained models, are released under an open-source license to foster community engagement and advance natural language processing research, particularly for two low-resource languages spoken by approximately 300 million people worldwide.
The deployment of Artificial intelligence (AI) in clinical speech applications has been limited in large part by the lack of interpretability, which is essential for establishing clinician trust and enabling effective decision support. Although methods such as SHapley Additive exPlanations (SHAP) aim to improve transparency in many clinical domains, their applicability to clinical speech-language pathology practice is uncertain. Since these methods rely on data modalities like acoustic signal features and spectrograms, which are unfamiliar to clinicians and misaligned with clinical workflows, the resulting interpretations may introduce additional burden and bias rather than provide clinically meaningful insight. To better understand this challenge, we conducted two consecutive user studies to systematically evaluate a commonly used SHAP-based interpretation design (a bar chart showing the influence of acoustic features on AI decisions) in dysarthria detection. Building on our prior works, eight factors were examined: faithfulness, computational efficiency, cognitive load, human-AI task performance, mental model, user trust, clinical understandability, and decision relevance. The results reveal a previously unrecognized risk in current interpretation practices. The seemingly intuitive bar-chart design frequently misled participating speech-language pathology (SLP) students to interpret feature influence as an indicator of clinical severity. Other findings include difficulty understanding AI mechanisms, discrepancies between human and model reasoning, and the limited ability of interpretations to address clinical questions. Through this work, we highlight the need for interpretation designs that are more closely aligned with clinical reasoning patterns and suggest practical considerations for developing speech-based AI systems that can be meaningfully integrated into clinical practice.
Generative Dialogue summarization involves extracting essential information from conversations and distilling it into succinct descriptions. This study introduces a novel multi-task learning framework that combines topic-based data augmentation and external knowledge generation for generative dialogue summarization. Our framework operates by annotating and enriching dialogue data based on topics at the data level, fostering varied expressions across different topics. Additionally, it integrates external knowledge at the entity and paragraph levels to help the model grasp common sense knowledge that is implicit in the context and enhance our model’s understanding of dialogue. Extensive experiments were carried out using two real-world datasets. The results demonstrate that our framework outperforms state-of-the-art models in dialogue summarization tasks, confirming the effectiveness of our proposed methodology.
This study investigates response time as a behavioral indicator related to listening effort (LE) for evaluating speech enhancement (SE) systems. English and Norwegian intelligibility matrix tests were conducted within a single-task paradigm that incorporated click-time recording (logging the precise time of all participant clicks), enabling simultaneous estimation of speech intelligibility and LE-related temporal behavior. Three temporal proxy measures for LE were examined—time per stimulus, reaction time, and word click time—across a broad range of input signal-to-noise ratios (SNRs) and for both discriminative and generative enhancement approaches. Time per stimulus showed an inverted-U pattern across SNRs, whereas reaction time and word click time exhibited monotonic behavior, providing more directly interpretable metrics for comparative evaluation. Analyses of pairwise SNR comparisons revealed that increases in our LE-related temporal measures at higher SNRs precede measurable intelligibility declines, suggesting that these temporal metrics can be more sensitive than intelligibility in this regime. Overall, the proposed framework—where LE-related measurements remain unknown to participants—offers a comprehensive and nuanced behavioral tool for SE evaluation, complementing intelligibility particularly under realistic, moderate-to-high SNR conditions.
Entrainment is a social adaptive mechanism which in human spoken interaction includes interlocutors unconsciously adjusting their vocal patterns and related behaviors to match those of their conversation partner. Entrainment offers important insights toward understanding the socio-cognitive characteristics of an individual. Quantifying entrainment patterns can also inform clinical diagnosis, long-term monitoring, and individualized interventions in neuro-developmental disorders characterized by deficits in communication and social interaction, such as Autism Spectrum Disorder (ASD). In this work, we model vocal entrainment in dyadic child-inclusive conversations to analyze behavioral traits of children with and without an autism diagnosis. Specifically, we explore contrastive-learning based unsupervised modeling to learn representations related to entrainment from speech features. We validate the proposed measures by using them to differentiate real conversations from simulated shuffled ones. Furthermore, we illustrate their utility in modeling various behaviors relevant to autism symptoms by correlation experiments and comparing the variation of the introduced measures in children under different demographic conditions.
In this paper, we propose a new Blind Source Separation (BSS) method for (under)-determined convolutive mixtures of speech signals based on Time-Frequency (TF) masking and recombination of observations, exploiting the sparsity of source signals in the TF domain. Unlike conventional TF-masking-based approaches that rely directly on observation vectors for clustering, the proposed method introduces a more robust feature based on the complex cosine similarity between observation vectors and a set of reference vectors. This leads to more reliable TF clustering and improved separation performance, particularly in challenging underdetermined scenarios. A key contribution of this work is the blind estimation of Relative Transfer Functions (RTFs) of the sources, without requiring any prior acoustic knowledge. Building on this, we propose a novel RTF-based recombination strategy that replaces the classical TF-masking reconstruction step. This significantly reduces the artifacts typically introduced by masking-based BSS methods. Our test results demonstrate that our method outperforms several existing TF-masking-based methods.
