The pursuit of a "unified" discrete token for both speech understanding and generation has led the Speech Language Model (SLM) community to heavily rely on Word Error Rate (WER) – the core metric for Whisper-style tokenizers – as the definitive proxy for representation quality. This fosters the assumption that low-WER tokens inherently preserve the information necessary for intelligible acoustic synthesis. We argue this is fundamentally deceptive. While high-frequency tokens succeed in generation tasks due to implicit information leakage, isolating pure semantic information at ultra-low frame rates strips away the finegrained articulation and micro-dynamics essential for ODE-based generation. Empirically validating this requires extreme compression without sacrificing WER – a methodological bottleneck, as standard fixed-stride downsampling arbitrarily truncates phonetic boundaries. To overcome this, we develop a dynamic compression tokenizer that intelligently aligns representations with semantic boundaries, achieving ultra-low frame rates with exceptionally low WER. Using these isolated "pure" semantic tokens, we expose the WER trap: when conditioning generative models – even with oracle duration alignments – the reconstructed speech suffers from severe articulation blur and is rendered acoustically unintelligible. Our findings demonstrate that semantic categorization rewarded by low WER is inherently orthogonal to the continuous phonetic trajectories required for synthesis, shattering the illusion of the unified token and advocating for explicitly decoupled speech representations.

There is a responsibility to advance automatic alcohol intoxication screening capabilities in modern automobiles to reduce the high rate of alcohol-related accidents and fatalities worldwide. Automatic speech-based alcohol intoxication screening offers a tremendous safety opportunity in the automotive industry due to its non-invasive convenience, comparatively inexpensive cost, and rapid result processing. Using the Alcohol Language Corpus (ALC), this study examines automatic alcohol intoxication classification based on participants' non-intoxicated/intoxicated omnimicrophone speech recordings. Experimentation of many different speech features (e.g., glottal, landmarks, linguistic, prosodic, spectral, syllabic, vocal tract coordination) across different blood alcohol concentration (BAC) ranges and specific verbal tasks show significant changes as participants' BAC increases. Intoxicated participants produce lower average fundamental frequency (F0) with an increase in F0 frequency modulation, breathiness and creakiness voice qualities in intoxicated recordings when compared to their non-intoxicated recordings. For the picture description and tongue twister tasks, manual irregularity disfluency and pause linguistic features significantly increase in intoxicated recordings. Further, for all verbal tasks, automatically extracted syllabic pause features show a significant increase in intoxicated recordings. Implementation of task-dependent support vector machine classifier model with a >= 0.001 BAC 'intoxication' sensitivity threshold increases alcohol classification by up to 8% absolute gain over a task-agnostic approach. Moreover, intoxication classification results demonstrate that taskdependent modeling with majority vote decision improves classification accuracy with up to 20% absolute gain depending on task when compared to file-by-file task-agnostic method results reported previously in ALC baseline studies that used higher quality headset microphone recordings.

Audio tokenization has emerged as a critical component in end-to-end audio language models, enabling efficient discrete representation learning for both audio understanding and generation tasks. However, existing audio tokenizers face fundamental limitations in understanding tasks due to single-modality constraints, particularly when audio signals contain ambiguous or incomplete information. While incorporating additional modality information can significantly enhance audio understanding, current multimodal fusion approaches invariably degrade reconstruction quality. This degradation is unacceptable for end-to-end audio systems that require high-fidelity audio generation capabilities. In this work, we investigate the root causes of reconstruction quality degradation in video-enhanced audio tokenization and present three key findings. First, the location of fusion within the tokenizer architecture is crucial for preserving reconstruction quality. Second, we show that contrastive learning, though effective in continuous representation fusion, is unsuitable for discrete tokenizers as it fails to enhance downstream task performance. Third, while feature-dimension fusion approaches achieve moderate success, we discover that fusing along the temporal axis – guided by the concept of distinctive features – yields significantly better results. Building on these insights, we introduce the Timing-Aware Pre-Quantization Fusion for Video-Enhanced Audio Tokenization, the first approach to successfully integrate visual information into audio tokenizer architectures while preserving reconstruction fidelity. Our approach not only maintains high-fidelity reconstruction but also achieves superior performance on downstream understanding tasks compared with audio-only tokenizers and established multimodal fusion baselines.

One technology adoption outcome is ever more frequent task interruptions, which makes interruption management a central human computer interface design problem. Studies on interruption often focused on the link between interruptions and genesis of error and eye gaze to validate associated task cues engagement. How eye behaviors reflect load type and level changes between primary and interrupting tasks and how emotional interruptions impact mental state in primary tasks - factors that contribute to cognitive processing - remain unanswered. In this study, we analyzed performance metrics and eye behaviors from 18 participants while they completed an uninterrupted and interrupted roleplay task. Our results reveal that although interruption notification could affect the time to attend to an interruption, high cognitive load resulted in a detrimental impact on completion time and low cognitive load led to time reset for primary tasks. No existing evidence suggests that affect type influences the time metrics, however, high arousal and valence induced by interruptions altered eye behavior upon returning to the primary task, indicating less visual information taken and under a high load level, which showed evidence of the impact of affective interruption for the first time. Noteworthily, the eye behaved differently at interruption beginning compared with the end and uninterrupted task transitions, indicating the feasibility of including eye behavior for interruption estimation. Overall, these results are the first experimental evidence for theories that interruption can cause high mental load, posit extra mental effort, forgetting primary tasks and behaviors being directed by the current most active goal.

Depression remains a pressing global mental health issue, driving considerable research into AI-driven detection approaches. While pre-trained models, particularly speech self-supervised models (SSL Models), have been applied to depression detection, they show unexpectedly poor performance without extensive data augmentation. Large Language Models (LLMs), despite their success across various domains, have not been explored in multi-modal depression detection. In this paper, we first establish an LLM-based system to investigate its potential in this task, uncovering fundamental limitations in handling multi-modal information. Through systematic analysis, we discover that the poor performance of pre-trained models stems from the conflation of high-level information, where one of the important reasons is that high-level features derived from both content and speech are mixed within pre-trained models model representations, making it challenging to establish effective decision boundaries. To address this, we propose an information separation framework that disentangles these features, significantly improving the performance of both SSL models and LLMs in depression detection. Our experiments validate this finding and demonstrate that the integration of separated features yields substantial improvements over existing approaches, providing new insights for developing more effective multi-modal depression detection systems.

Large Language Models (LLMs) have demonstrated remarkable success across diverse fields, establishing a powerful paradigm for complex information processing. This has inspired the integration of speech into LLM frameworks, often by tokenizing continuous audio via neural speech codecs, enabling powerful speech language models. However, this dominant tokenization strategy relies on uniform frame-based processing at fixed time intervals. This fixed-rate approach, while effective for linguistic content, destroys the temporal dynamics. These dynamics are not noise but are established as primary biomarkers in clinical applications such as depression detection. To address this gap, we introduce the Distinctive Feature Codec (DFC), an adaptive framework engineered to preserve this vital timing information. Drawing from linguistic theory, DFC abandons fixed-interval processing and instead learns to dynamically segment the signal at perceptually significant acoustic transitions. This generates variable-length tokens that efficiently encode the temporal structure. As a key contribution, this work is the first to integrate traditional distinctive features into a modern deep learning codec for a temporally sensitive task such as depression detection. We also introduce the Group-wise Scalar Quantization (GSQ) approach to stably quantize these variable-length segments. Our distinctive feature-based approach offers a promising alternative to conventional frame-based processing and advances interpretable representation learning in the modern deep learning speech depression detection framework.

The Mamba-based model has demonstrated outstanding performance across tasks in computer vision, natural language processing, and speech processing. However, in the realm of speech processing, the Mamba-based model's performance varies across different tasks. For instance, in tasks such as speech enhancement and spectrum reconstruction, the Mamba model performs well when used independently. However, for tasks like speech recognition, additional modules are required to surpass the performance of attention-based models. We propose the hypothesis that the Mamba-based model excels in "reconstruction" tasks within speech processing. However, for "classification tasks" such as Speech Recognition, additional modules are necessary to accomplish the "reconstruction" step. To validate our hypothesis, we analyze the previous Mamba-based Speech Models from an information theory perspective. Furthermore, we leveraged the properties of HuBERT in our study. We trained a Mamba-based HuBERT model, and the mutual information patterns, along with the model's performance metrics, confirmed our assumptions.

In the speech signal, acoustic landmarks identify times when the acoustic manifestations of the linguistically motivated distinctive features are most salient. Acoustic landmarks have been widely applied in various domains, including speech recognition, speech depression detection, clinical analysis of speech abnormalities, and the detection of disordered speech. However, there is currently no dataset available that provides precise timing information for landmarks, which has been proven to be crucial for downstream applications involving landmarks. In this paper, we selected the most useful acoustic landmarks based on previous research and annotated the TIMIT dataset with them, based on a combination of phoneme boundary information and manual inspection. Moreover, previous landmark extraction tools were not open source or benchmarked, so to address this, we developed an open source Python-based landmark extraction tool and established a series of landmark detection baselines. The first of their kinds, the dataset with landmark precise timing information, landmark extraction tool and baselines are designed to support a wide variety of future research.

The automatic identification and analysis of pronunciation errors, known as Mispronunciation Detection and Diagnosis (MDD) plays a crucial role in Computer Aided Pronunciation Learning (CAPL) tools such as Second-Language (L2) learning or speech therapy applications. Existing MDD methods relying on analysing phonemes can only detect categorical errors of phonemes that have an adequate amount of training data to be modelled. With the unpredictable nature of the pronunciation errors of non-native or disordered speakers and the scarcity of training datasets, it is unfeasible to model all types of mispronunciations. Moreover, phoneme-level MDD approaches have a limited ability to provide detailed diagnostic information about the error made. In this paper, we propose a low-level MDD approach based on the detection of speech attribute features. Speech attribute features break down phoneme production into elementary components that are directly related to the articulatory system leading to more formative feedback to the learner. We further propose a multi-label variant of the Connectionist Temporal Classification (CTC) approach to jointly model the non-mutually exclusive speech attributes using a single model. The pre-trained wav2vec2 model was employed as a core model for the speech attribute detector. The proposed method was applied to L2 speech corpora collected from English learners from different native languages. The proposed speech attribute MDD method was further compared to the traditional phoneme-level MDD and achieved a significantly lower False Acceptance Rate (FAR), False Rejection Rate (FRR), and Diagnostic Error Rate (DER) over all speech attributes compared to the phoneme-level equivalent.

Cognitive impairment (CI) is of growing public health concern, and early detection is vital for effective intervention. Speech has gained attention as a non-invasive and easily collectible biomarker for assessing cognitive decline. Traditional CI detection methods typically rely on supervised models trained on acoustic and linguistic features extracted from speech, which often require manual annotation and may not generalise well across datasets and languages. In this work, we propose the first zero-shot speech-based CI detection method using the Qwen2- Audio AudioLLM, a model capable of processing both audio and text inputs. By designing prompt-based instructions, we guide the model in classifying speech samples as indicative of normal cognition or cognitive impairment. We evaluate our approach on two datasets: one in English and another multilingual, spanning different cognitive assessment tasks. Our results show that the zero-shot AudioLLM approach achieves performance comparable to supervised methods and exhibits promising generalizability and consistency across languages, tasks, and datasets.

Transformer and its derivatives have achieved success in diverse tasks across computer vision, natural language processing, and speech processing. To reduce the complexity of computations within the multi-head self-attention mechanism in Transformer, Selective State Space Models (i.e., Mamba) were proposed as an alternative. Mamba exhibited its effectiveness in natural language processing and computer vision tasks, but its superiority has rarely been investigated in speech signal processing. This paper explores solutions for applying Mamba to speech processing by discussing two typical speech processing tasks: speech recognition, which requires semantic and sequential information, and speech enhancement, which focuses primarily on sequential patterns. The experimental results confirm that bidirectional Mamba (BiMamba) consistently outperforms vanilla Mamba, highlighting the advantages of a bidirectional design for speech processing. Moreover, experiments demonstrate the effectiveness of BiMamba as an alternative to the self-attention module in the Transformer model and its derivates, particularly for the semantic-aware task. The crucial technologies for transferring Mamba to speech are then summarized in ablation studies and the discussion section, offering insights for extending this research to a broader scope of tasks.

Mental state induced by different task contexts and load levels is of interest for human health and wellness, and eye activity extracted from infrared eye images is well-suited to estimate it. As a useful tool for emotion recognition, facial action units (FAUs) extracted from facial images are well-established, however these are insufficiently detailed for the eye. In this paper, we extract discrete eye behaviors from eyelid, iris and pupil boundaries and propose eye action units (EAUs) based on the eye appearance (behavior), providing a detailed and interpretable representation that shares the advantages of FAUs and describes the wide range of eye states and shapes. Eight volunteers annotated 11 EAUs for 120 eye images, represented by a series of discrete eye behaviors. Analysis shows that the EAUs can be viably characterized by fundamental eye behaviors with moderate to substantial agreement. When evaluating discrete eye behaviors and EAUs for recognition of four mental states and two load levels, the former achieved significantly higher accuracy than conventional features of pupil size change and blink rate, especially using behavior duration features, and EAUs outperformed combinations of discrete eye behaviors in general, implying their utility as an action unit.

Accurately describing the tasks that comprise a day in your life is an inherently multimodal characterisation: after beginning each task, you become loaded to some extent by the objects, movements, communication and/or mental challenges required by that task, then you later switch to a new task, and so on. Automating task analysis of this kind, which to date is manual, post-hoc and subjective, has proven challenging. Wearable systems provide the helpful opportunity to position heterogeneous non-invasive sensors directly where they are most useful for task analysis: on the head and close to the eyes and mouth. In a longitudinal context, continuous data acquisition from these sensors is typically inefficient because the most interesting changes tend to occur infrequently, and there is scope for more investigation of automatic event-based analysis approaches inspired by the way humans annotate multimodal data. This presentation focuses on key research problems and recent results spanning psychophysiological motivations, feature extraction, including the accurate extraction of eye action units from very small near-field infrared cameras mounted on glasses frames, feature variability, multimodal fusion, machine learning, and system design for continuous and robust automatic task analysis from wearable sensors. It also highlights examples of how multimodal analysis of interpretable event-based approaches can yield new insights and new machine learning research directions. Task analytics of this kind represent huge potential for individual users to empower themselves and interact more seamlessly with machines.

Large Language Models (LLMs) have been increasingly adopted for health-related tasks, yet their performance in depression detection remains limited when relying solely on text input. While Retrieval-Augmented Generation (RAG) typically enhances LLM capabilities, our experiments indicate that traditional text-based RAG systems struggle to significantly improve depression detection accuracy. This challenge stems partly from the rich depression-relevant information encoded in acoustic speech patterns information that current text-only approaches fail to capture effectively. To address this limitation, we conduct a systematic analysis of temporal speech patterns, comparing healthy individuals with those experiencing depression. Based on our findings, we introduce Speech Timing-based Retrieval-Augmented Generation, SpeechT-RAG, a novel system that leverages speech timing features for both accurate depression detection and reliable confidence estimation. This integrated approach not only outperforms traditional text-based RAG systems in detection accuracy but also enhances uncertainty quantification through a confidence scoring mechanism that naturally extends from the same temporal features. Our unified framework achieves comparable results to fine-tuned LLMs without additional training while simultaneously addressing the fundamental requirements for both accuracy and trustworthiness in mental health assessment.

A day in your life can be described in terms of a series of tasks: you begin each task, become loaded to some extent by the objects, movements, communication and/or mental challenges required by that task, then you later switch to a new task, and so on. However, methods for task analysis to date, such as the diary method are manual, post-hoc and subjective, and therefore impossible to scale for large numbers of users or to directly interface with computing systems. Recent developments in wearable sensing promise change, with the opportunity to position non-invasive sensors directly where they are most useful for task analysis: on the head and close to the eyes and mouth. Head-mounted wearable sensors open up the prospect of continuous automatic analysis of physical and mental tasks and behaviour based on non-invasive signals such as eye activity, speech and body movement. Although surprisingly challenging to accurately extract, pupil size, saccade and blink can be continuously estimated from very small near-field infrared cameras, mounted on glasses frames. Similarly eyewear of this kind are ideal for mounting inertial measurement units, microphone arrays and other sensors that can continuously estimate task load, movement and communication, even in sedentary settings. In a longitudinal context, continuous data acquisition is typically inefficient because the most interesting changes tend to occur infrequently, and this work investigates automatic event-based analysis approaches inspired by the way humans annotate multimodal affective data. This presentation focuses on key research problems and recent results, spanning the psychophysiological basis, feature extraction, feature variability, machine learning, and system design for continuous and robust automatic task analysis from wearable sensors. Task analytics of this kind represent huge potential for individual users to empower themselves and interact more seamlessly with machines.

Human task transition detection has previously focused on body movement signals and corresponding statistical features, however, these may be less effective for detecting transitions between mental tasks. Eye features, which have been shown effective in cognitive task processing, have rarely been adopted in physical task analysis or activity transition, and are investigated herein. It is also of interest to tokenize time series signals into meaningful word-like event-based features, which provides a novel and more interpretable alternative to statistical features and access to timing information between events, which has the potential to capture human activity patterns but has not been investigated for task transition detection. In this paper, we combine head movement and eye features and analyze their contributions; apply event-based features to recognize task transition detection using natural language processing methods; develop novel event-based features to capture the change in pupil size; represent the timing information between consecutive events; and reveal insights into event-based and timing features. Evaluations on a novel semi-naturalistic dataset show that automatic task transition can be quite reliably detected using head movement and eye activity from wearable sensors, that the classification performance of fused event-based and timing features is comparable with that of statistical features, and for eye features only, both the proposed event-based features and timing features significantly outperform the statistical features. The fusion of event-based features and timing features also slightly improves the classification performance over systems based on either feature set alone.

Mobile technologies have become integral to daily life, and understanding users’ emotional states during interactions is crucial for enhancing user experience. However, integrating affective perception, behavior analysis, and affective computing for mobile technologies presents multifaceted challenges, ranging from technological limitations to ethical considerations. This workshop proposes a collaborative exploration of cutting-edge solutions for affective computing for mobile technologies. We aim to bring together experts to explore topics such as: user behavior analytics, user experience design, affective computing applications, cultural and contextual considerations, and the ethical implementation of affective computing. This workshop aims to bring together researchers and practitioners from both academia and industry to identify and explore: 1) innovative solutions, 2) novel applications, and 3) key challenges in this area to drive research in the coming decade. The long-term goal is to create a strong interdisciplinary research community that includes researchers and practitioners from HCI, HRI, Ubiquitous Computing, Cognitive Psychology, Mobile Technology, Interaction Techniques, User Privacy, and Design. We envision ongoing research collaborations and accelerating innovations in affective computing for mobile technologies.

Depression is a critical concern in global mental health, prompting extensive research into AI-based detection methods. Among various AI technologies, Large Language Models (LLMs) stand out for their versatility in mental healthcare applications. However, their primary limitation arises from their exclusive dependence on textual input, which constrains their overall capabilities. Furthermore, the utilization of LLMs in identifying and analyzing depressive states is still relatively untapped. In this paper, we present an innovative approach to integrating acoustic speech information into the LLMs framework for multimodal depression detection. We investigate an efficient method for depression detection by integrating speech signals into LLMs utilizing Acoustic Landmarks. By incorporating acoustic landmarks, which are specific to the pronunciation of spoken words, our method adds critical dimensions to text transcripts. This integration also provides insights into the unique speech patterns of individuals, revealing the potential mental states of individuals. Evaluations of the proposed approach on the DAIC-WOZ dataset reveal state-of-the-art results when compared with existing Audio-Text baselines. In addition, this approach is not only valuable for the detection of depression but also represents a new perspective in enhancing the ability of LLMs to comprehend and process speech signals.

BACKGROUND:Inconsistent observations of pupillary response and blink change in response to different specific tasks raise questions regarding the relationship between eye measures, task types and working memory (WM) models. On the one hand, studies have provided mixed evidence from eye measures about tasks: pupil size has mostly been reported to increase with increasing task demand while this expected change was not observed in some studies, and blink rate has exhibited different trends in different tasks. On the other hand, a WM model has been developed to integrate a component to reconcile recent findings that the human motor system plays an important role in cognition and learning. However, how different tasks correlate with WM components has not been experimentally examined using eye activity measurements.AIMS:The current study uses a four-dimensional task load framework to bridge eye measures, task types and WM models.SAMPLE:Twenty participants (10 males, 10 females; Age: M = 25.8, SD = 7.17) above 18 years old volunteered. All participants had normal or corrected to normal vision with contact lenses and had no eye diseases causing obvious excessive blinking.METHODS:We examined the ability of pupil size and blink rate to index low and high levels of cognitive, perceptual, physical and communicative task load. A network of the four load types and WM components was built and analysed to verify the necessity of integrating a physical task-related component into the WM model.RESULTS:Results demonstrate that pupil size can index cognitive load and communicative load but not perceptual or physical load. Blink rate can index the level of cognitive load but is best at discriminating perceptual tasks from other types of tasks. Furthermore, pupil size measurement of the four task types was explained better during structural and factor analysis by a WM model that integrates a movement-related component.CONCLUSIONS:This research provides new insights into the relationship between eye measures, task type and WM models and provides a comprehensive understanding from which to predict pupil size and blink behaviours in more complex and practical tasks.