Multimodal large language models (MLLMs) are increasingly being deployed in emotion-sensitive interactions such as mental health support. When visual, textual, and audio cues are inconsistent, models may form a one-sided understanding based on superficial cues, leading to a misinterpretation of the user's emotional state. This paper investigates the internal representations of multimodal affective conflicts at the last conditioning token, as well as the relationship between pre-generation states and subsequent affective misreads. We extract the full-layer trajectories of unimodal and joint inputs, learn state representations via a Trajectory Manifold Encoder, and calculate state indices to identify state patterns. Experimental results demonstrate that conflicting samples and aligned samples exhibit distinguishable state structures, which correlate with affective misreads. The trajectory manifold representations outperform baselines in terms of representation quality, cross-domain transferability, and performance under limited conflict supervision. The State-Guided Response reduces affective response errors and enhances overall response quality. This paper provides an early, interpretable analytical approach to understanding the relationship between multimodal affective conflicts and misreads.
Decoding emotional arousal from Electrodermal Activity (EDA) is a critical task in affective computing. However, existing approaches are predominantly based on data-driven ”black box” deep learning models, which lack quantitative interpretability with respect to physiological mechanisms. Meanwhile, although traditional EDA inversion methods can infer sympathetic neural activity, they are unable to explicitly characterize its relationship with psychological variables. To address these limitations, we propose a Physiology-Physics-Psychology Informed Neural Network (P $^{3}$ INN), which embeds a secondorder differential equation based on the pore valve model as a physical constraint within a CNN-LSTM hybrid architecture to accurately invert latent sympathetic neural activity from EDA signals. Furthermore, it explicitly captures the nonlinear relationship between neural activity and emotional arousal by introducing invertible mathematical mapping functions, which enable end-to-end joint optimization of the entire framework. Experimental results on the CASE and CEAP-360VR datasets demonstrate that P $^{3}$ INN significantly outperforms traditional EDA decomposition methods and state-of-the-art deep learning models in emotional arousal decoding performance. Further analysis reveals that a logarithmic mapping best captures the psychological-neural transduction mechanism, and the model parameters uncover significant gender-related physiological differences in neural drive sensitivity. This study not only improves the accuracy of arousal decoding but also provides a highly interpretable modeling paradigm for computational psychophysiology.
Emotional well-being is essential for healthy aging, yet objective behavioral measures of music-evoked emotion in older adults remain limited. Existing affective computing datasets primarily feature young adults and acted expressions, offering limited insight into the subtle, naturalistic behaviors that characterize late-life emotional engagement. Here we introduce a multimodal dataset of older adults (ages 60+) recorded during a controlled music-listening protocol designed to evoke a broad range of affective responses. The dataset contains recordings from 17 participants, yielding 200 valid clips of approximately 30 seconds each ($\approx$183,220 frames in total). For each of twelve music clips, participants provided self-reports of valence, arousal, liking, and familiarity. A board-certified music therapist annotated each session with detailed descriptions of facial expressions, head and body movements, attentional behaviors, and inferred mood dynamics. De-identified computer-vision features, including facial action units, head pose, gaze direction, and upper-body keypoints, were extracted using OpenFace and OpenPose. This dataset provides a comprehensive resource for examining music-induced emotion, micro-behavior dynamics, and therapist-guided affective interpretation in older adults. It serves as a foundation for developing computational models for emotion recognition, behavior understanding, and AI-assisted assessment tools in aging and dementia care.
Accurate estimation of cognitive workload is critical for understanding and optimizing human performance in complex task environments. In this paper, we propose a hierarchical latent cognitive model that is a probabilistic modeling framework combining Latent Class Analysis and Bayesian networks to estimate cognitive workload at a task-level. Our approach leverages multimodal data streams to infer intermediate latent constructs; specifically engagement and drive states, which subsequently inform workload estimation. We systematically evaluate model performance through an ablation study and comparison with established machine learning classifiers, demonstrating superior accuracy and predictive consistency of our hierarchical latent approach. Moreover, extensive interpretability analyses, including structural visualization and conditional parameter evaluations, illustrate the model's capacity to transparently represent internal cognitive states and their relationships to observed behaviors. These results highlight the effectiveness of our proposed model for both accurate workload estimation and meaningful interpretability, enhancing transparency, and generalizability for diverse practical applications.
Stress and mental workload (MWL) frequently co-occur and share physiological signatures, reducing discriminability and limiting cross-subject generalisation. To address this, we implemented an immersive virtual reality (VR) adaptation of the Trier Social Stress Test (TSST) using a 2×2 factorial design that manipulated psychosocial stress (high/low) and MWL (high/low), enabling controlled evaluation under co-occurrence. We recorded electroencephalography (EEG), electrodermal activity, heart rate and heart rate variability, and pupillometry in 44 participants. Classification was used not as a methodological contribution or classifier benchmark, but as a probe of cross-subject physiological discriminability under co-occurring stress and MWL. Using leave-one-subject-out cross-validation, MWL was classified above chance across stress levels, whereas stress did not generalise beyond chance despite robust subjective effects. EEG spectral ratio features showed greater stability and stronger MWL discriminability than absolute power measures, while autonomic features showed limited state specificity. These findings indicate that physiological discriminability under co-occurring demands is state dependent. Under immersive, socially evaluative conditions, MWL exhibits modest but generalisable neural features across individuals, whereas stress appears substantially more sensitive to contextual and individual variation. This work clarifies the conditions under which cross-subject physiological generalisation is feasible and informs the evaluation and modelling of co-occurring cognitive-affective states in realistic settings.
Music has a profound power to induce and regulate emotional states. Inspired by the iso principle, in which listeners are guided towards a desired emotional state by first matching their current mood through music then gradually shifting the music towards the intended state, we explore the use of an AI-based automatic music generation (AI-AMG) system AffectMachine to support emotion regulation. We conducted a study with 54 participants and two versions of AffectMachine, Pop and Classical, to evaluate their ability to guide listeners towards calm and happy emotional states. Based on self-reported measures of emotion (arousal and valence), both versions of the AffectMachine successfully guided the majority of participants towards the target emotions, with stronger and more consistent effects on arousal than on valence. Interestingly, the high-arousal target emotion, Happy, tended to be induced more quickly than the low-arousal target emotion, Calm. We also found strong evidence of a positive correlation between change in arousal and change in heart rate. Our findings provide early evidence that AI AMG systems implementing the iso principle can support affect modulation, and offer insights for the design of adaptive music systems for therapeutic applications.
Multimodal large language models (MLLMs) have shown strong performance in generic visual understanding, yet their ability to perceive and reason about emotions in fine art remains underexplored. Unlike natural images, artworks frequently convey emotion indirectly through composition, color, style, and cultural context rather than explicit objects or events. To solve this, we proposed Art Emotion Reasoning Benchmark (AER-Bench), a benchmark for evaluating affective cognition over artworks with an explicit progression from low-level visual affect cues to high-level emotional reasoning. Specifically, we construct a structured artwork annotation schema and convert it into deterministic question templates spanning closed-ended perception, emotion ranking, dimensional affect estimation, open-ended description, and reasoning-oriented explanation. Using AER-Bench, we evaluate representative open-source and proprietary MLLMs with both objective metrics and rubric-based judging. Our results reveal a consistent gap between affective label prediction and evidence-grounded interpretation: Current MLLMs often produce plausible affective outputs, but struggle to ground them in perceptual, comparative, and contextual evidence. AER-Bench therefore provides a practical and extensible testbed for diagnosing and advancing emotionally aware visual intelligence.
Multimodal Emotion-Cause Pair Extraction (MECPE) aims to identify emotion utterances and their corresponding cause utterances in multi-party conversations. Although pre-trained models have delivered promising results, many existing methods are still developed primarily from an engineering perspective. Consequently, they tend to overlook the linguistic regularities that govern how emotion propagates in conversations. Emotion propagation in conversations is inherently unidirectional along the temporal axis, while its surface realization is further influenced by heterogeneity across modalities. We therefore characterize MECPE as facing two key challenges: 1) how to model the temporal dependencies of emotion propagation, and 2) how to capture both the complementary and inconsistent aspects of multimodal emotion features. To address these challenges, we propose a dynamic temporal differential multimodal emotion filtering model (DynTD). To encode the temporal propagation patterns of emotion in conversations, we introduce a Temporal Differential Table Fusion (TDTF) algorithm, which constructs an emotion cause table via a first-order temporal difference operator, explicitly modeling the unidirectional propagation and evolution of emotion over time. To account for the differences and complementarity in emotional expression across modalities, we further design a cross-modal Dynamic Gated Attention Filtering (DGF) mechanism. Using text as the primary modality, DGF employs cross-attention to extract complementary emotion related cues from the audio and video, while gated units suppress modality-induced discrepancies in emotional expression, thereby enabling adaptive fusion of multimodal emotion information. Experiments on several benchmark datasets demonstrate the superiority of DynTD, and extensive ablation studies confirm the effectiveness of conversational temporal modeling and multimodal discrepancy filtering for emotion analysis in conversations.
Understanding the psychological mechanisms underlying individual participation is essential for modeling information propagation dynamics on social media platforms. As a core psychological construct, personality significantly influences individuals' affective responses and behavioral decisions. However, existing propagation models often treat users as homogeneous actors and fail to sufficiently capture how personality traits shape users' affective participation processes, including commenting behavior, state transitions, and cross-platform propagation. To address this limitation, this study proposes the MP-SL3CI (Multi-platform Susceptible–Latent–3Commenting–Immune) model, which incorporates personality traits into a two-layer coupled network to examine dissemination on multi-layer platforms. Users are categorized into emotional, rational, and apathetic types to analyze personality effects on commenting behavior and cross-platform migration. Personality traits are extracted from user comments using pre-trained models, and simulations with real-world communication data are conducted to more realistically reproduce information dissemination processes across multi-layered networks. Numerical analyses, including comparative and ablation experiments, demonstrate that incorporating personality traits and multi-platform interaction mechanisms significantly improves the model's explanatory capacity and yields superior fitting performance compared with baseline models. Sensitivity analysis further identifies key factors influencing user decisions and cross-platform migration, providing theoretical support for propagation dynamics research and communication strategy design.
Music-related emotion recognition (MRER) aims to automatically predict emotional states based on different musical forms (e.g., lyrics, music, and music videos). Despite notable advancements in the field, MRER still faces the following challenges: 1) heterogeneity in emotion representations across datasets, including categorical and dimensional labels; 2) scarcity of comprehensive modalities and emotion annotations (e.g., static and dynamic) in existing music video emotion datasets; and 3) absence of a benchmark that contains various tasks and evaluation metrics for MRER. In this paper, we first propose a unified emotion representation consisting of emotion category and intensity, along with correlated conversion strategies to integrate disparate labels. Building upon the unified representation, we introduce an innovative emotion annotation framework MVAnno, which employs a hierarchical continual fine-tuning process on multiple modalities to obtain accurate emotion annotations. We also construct a large-scale music video emotion dataset MVEmo, which comprises 11 K samples, with 11 K static emotion labels and 5 M dynamic emotion labels. Finally, we present MVEmo-Bench, a multi-task benchmark with evaluation metrics specifically designed for MRER tasks and the natural language outputs generated by large language models (LLMs). Our work makes a significant contribution to MRER by addressing key challenges and providing robust foundations for future research.
Harmful memes are widely spread across the Internet, creating an urgent need for effective detection. Memes often embed cultural metaphors and social biases within concise visual–textual expressions. This high degree of semantic compression makes fine-grained harmful meme detection particularly challenging. Existing approaches either rely on large-scale annotated data or depend on large multimodal models for computationally expensive rationale generation, limiting scalability and generalization. To address these challenges, we propose DeMeme, a semantic decompression framework designed for lightweight adaptation and interpretability. In the first stage, the Risk-Aware Routing module efficiently estimates potential harmfulness using semantic anchors, routing high-risk memes to the retrieval stage for further interpretation. In the second stage, the Semantic Decompression Retrieval module retrieves high-similarity “Term–Definition” pairs, reconstructing hidden harmful semantics into explicit cultural and emotional cues. Experiments on three English and Chinese benchmarks show that the training-free Retrieve-All variant achieves strong zero-shot performance. These results confirm the effectiveness and cross-cultural adaptability of semantic decompression for fine-grained harmful meme understanding.
Affective art interaction provides a low-threshold way for users to externalize and regulate their emotions through creative activities. Current intelligent affective art interaction systems predominantly rely on text- or audio-based guidance, successfully assisting users in expressing their emotions. However, these systems often fall short in providing sufficient social companionship, visual sensing, and embodied feedback throughout the creative process. To address this, we present a 3D embodied agent-mediated affective art interaction system. Rather than treating the agent as a general embodied conversational assistant, the proposed system centers on an Affective-Art Composition Game, where users express and reshape affect through interaction and agent-guided art design. The system features two main components: an Affective-Art Composition Game and an affective conversational agent with a customizable 3D avatar providing socially present and embodied feedback. To address insufficient implicit affect capture, we introduce a semantic-guided video emotion perception module that contributes facial-video evidence to user-state understanding. To improve the avatar's behavioral feedback, we introduce a structure-aware avatar rigging module that supports stable embodied expression. On the MAFW benchmark, the proposed emotion-perception module achieves 44.45% UAR and 58.67% WAR, while the rigging module reduces joint-localization and skinning errors. User evaluations indicate that enhanced sensing improves perceived empathy and that improved rigging strengthens social presence and affinity. Collectively, this paper shows the potential of 3D embodied agent-mediated affective art interaction.
With the rise of social media, Multimodal Sentiment Analysis has gained growing attention. However, existing approaches still face three major challenges: (i) how to stabilize modality-specific representations and reduce the influence of inherent noise; (ii) how to enhance intra-modal discriminability so as to effectively preserve the heterogeneity of different modalities during fusion; and (iii) how to balance cross-modal semantic conflicts while fully leveraging the textual modality information. To address these issues, we propose a Semantic Purification and Time-Causal Stabilization with Energy-Inspired Arbitration Framework (SCEF). SCEF consists of three key modules: (i) the Semantic Causal Purification and Stabilization Module (SCPS) operates from both semantic and causal perspectives to effectively suppress inherent noise in textual representations, enhance their dynamic stability, and strengthen their intra-modal discriminability; (ii) the Causal Time-Frequency Heterogeneous Enhancement Module (CTHE) employs time-causal multi-scale time-frequency modeling to suppress the inherent noise of the visual and acoustic modalities while enhancing their structural characteristics and intra-modal discriminability; and (iii) the Energy-Inspired Recalibration and Arbitration Module (ERA) processes cross-modal conflicts through energy-driven recalibration and arbitration strategies, and implicitly strengthens the role of the textual modality via Stochastic Modality Dropout. Extensive cross-lingual experiments show that SCEF achieves superior performance across multiple benchmarks, reaching 90.85% Acc-2 on the CMU-MOSI dataset. The source code is publicly available at: https://github.com/CICC-XJTLU/SPTCS-EIA
Automated depression detection, which aims to assess the severity of a subject's depression, has witnessed rapid advancements in recent years yet faces persistent challenges. First, existing models predominantly focus on static facial features or short-term expressions, failing to adequately model psychomotor retardation, the phenomenon of delayed or slowed responses to stimuli. This cardinal clinical sign of depression involves inhibited temporal dynamics that are crucial for diagnosis. The omission of these dynamics critically limits the ability of current models to capture latent response patterns, such as reaction latency, that are essential for distinguishing clinical depression from transient sadness. Second, the generalization performance of current models is frequently suboptimal. Due to the limited scale of available datasets, these models tend to confound identity-specific or other non-depressive facial characteristics with key depressive indicators, thereby impairing their generalization capabilities. To address these limitations, we propose a novel Depression Feature Enhancement Module (DFEM). This module leverages a delay-aware proxy to capture slowed temporal dynamics while fusing long-term cues with short-term dynamics to enhance feature discriminability. Furthermore, to bolster generalization, we introduce an Identity-aware Prototypical Decoupling Regularized Module (IPDRM). This mechanism imposes a structured organization on identity-specific information within the feature space, implicitly compelling the network to encode depression-relevant cues in the residual feature variations, significantly enhancing model robustness. Extensive experiments on multiple benchmark datasets demonstrate that our proposed model achieves state-of-the-art performance and exhibits superior generalization capabilities. The code is publicly available at https://github.com/code-jiaqi/PRADD.
As a spontaneous non-intrusive biometric cue, gait conveys reliable affective information. However, the subtle and fine-grained motion of joints makes gait emotion recognition more challenging than general emotion recognition, particularly under the constraints of dataset scarcity and the high complexity of extracting discriminative cues from subtle motions. While existing models demonstrate competitive performance, they often do not account for the inherent heterogeneity of the dataset. Specifically, data from different acquisition devices exhibit clear cross-modality variations, which complicate the development of a unified emotion representation. In this work, we present CTR-Gait, a single-stream network designed for semantic-aware and fine-grained gait emotion recognition. The backbone integrates Channel-wise Topology Refined graph convolutions with Temporal Transformers to capture dynamic spatial correlations and global temporal dependencies. To structure the affective feature space, we introduce Emotion-Semantic Contrastive Learning (ESCL), which utilizes learnable Valence-Arousal centers to achieve semantically consistent emotion clustering. Furthermore, to tackle the inherent ambiguity in distinguishing subtle negative emotions, we design a specialized SadExpert module that dynamically refines predictions for the minority class. With 3.76 M parameters and 0.56 GFLOPs, CTR-Gait achieves 92.66% accuracy on the Emotion-Gait benchmark, surpassing existing state-of-the-art methods. At the same time, we conducted UAV evaluations of the field sequences with the model, and the results demonstrate the model's potential for practical deployment.
Micro-expressions (MEs) are spontaneous, subtle facial movements that reveal genuine emotions, making them indispensable for high-stakes applications like security and clinical diagnosis. However, Micro-Expression Recognition (MER) is currently hindered by a dual bottleneck: the intrinsic difficulty of extracting discriminative features from subtle motions and the extrinsic privacy constraints that restrict centralized training in real-world scenarios. To address these challenges, we propose FED-PsyAU, a unified framework that synergizes psychological insights with privacy-preserving computing. First, addressing the feature representation challenge, we propose a psychological study, which quantifies the coordination and incoordination between upper and lower facial Action Units (AUs). Leveraging these priors, we develop a Dynamic Prior Knowledge-based Graph Attention Network (DPK-GAT) combined with a novel Spatial Position-Aware Transformer (SPAT) and an Expression-Specific Attention (ESA) module. This architecture effectively integrates structural psychological constraints with data-driven statistical patterns to learn hierarchical facial motion features. Furthermore, to tackle data privacy and scarcity, we integrate this model into a Personalized Federated Learning (P-FedProx) framework, enabling collaborative training across distributed clients without raw data exchange. Extensive experiments on standard ME databases demonstrate that our method achieves state-of-the-art performance in both centralized and federated settings, validating the efficacy of incorporating psychological topology into privacy-preserving systems. Our implementation is publicly available at https://github.com/MELABIPCAS/FED-PsyAU.git.
Facial Action Units (FAUs) capture the micro-level facial movement patterns of individuals during emotional expression, emotional response, and social interaction processes. While effectively preventing the leakage of personal identity information, they provide a robust representation for depression recognition with the added advantage of privacy protection. However, existing studies lack the guidance of external emotional stimuli and fail to adequately model interpretability. To address this, we have constructed the Emotional Image Viewing Experiment (EIVE) dataset for depression recognition based on FAUs. Meanwhile, We present SFENet, a shapelet-based facial expression interpretability model for depression recognition. Specifically, we extract shapelets for each FAU from subject samples using perceptually important points and construct shapelet-based initialization convolution kernels. Building upon this kernel, we propose a shapelet-convolution-based historical information enhancement module that strengthens feature extraction through historical information aggregation and time-scale interaction. Additionally, we design a sliding-window-based shapelet-sample similarity loss to update shapelet features, optimizing model training by measuring the alignment between features and sample sequences. Interpretability is derived from the parameters initialized by the shapelet convolution kernel and fitted through model training. SFENet achieves strong competitive performance in both EIVE and AVEC 2014, revealing that the case group exhibits a stronger tendency for negative facial expression unit activation under different emotional conditions. Our work is available at https://github.com/yjy-97/SFENet.
In scenarios with missing modalities, although existing multimodal sentiment analysis has made significant progress in handling incomplete modalities, there remains room for improvement in modality feature learning and cross-modal alignment. To better bridge the semantic gap between different modalities and reduce reconstruction errors for missing modalities, A multimodal sentiment analysis model based on information enhancement toward modal absence, referred to as IETMB, was proposed. The model is designed to fully capture modality-invariant features through a six-branch cross-modal Transformer and contrastive learning, while encoding modality-specific features to reduce modality distribution gaps and fully utilize multimodal information. Then, a cascaded attention self-encoder-based feature reconstruction module is designed to generate robust reconstructed features by enhancing the feature representation step-by-step with multi-level attention self-encoders. Finally, the residual information enhancement module introduces a residual component in order to enhance the semantic information of the reconstructed features to generate semantically rich fusion features. Experimental results on the public datasets IEMOCAP, MSP-IMPROV, and CMU-MOSI show that the model improves in performance compared with the advanced models. The weighted accuracy (WA) improved by 0.57% and 0.33% in IEMOCAP and MSP-IMPROV, respectively, and the mean absolute error (MAE) decreased by 0.0069 on CMU-MOSI.
Reliable recognition of human emotions from multimodal signals is important for human-computer interaction and affective intelligence systems. However, multimodal sentiment analysis remains challenged by heterogeneous modality representations, insufficient modeling of high-order interactions, and limited generalization under distribution shifts. To address these issues, this paper proposes SHF-MSA, a hypergraph-based multimodal sentiment analysis model that integrates representation coordination, structured high-order fusion, and multi-task learning. First, a Unified Pre-encoding Layer (UPL) employs modality-specific encoding, shared-space projection, and statistical normalization to reduce representation discrepancies among textual, acoustic, and visual modalities while preserving modality-specific and temporal information. Next, a Temporal Synergistic Hypergraph Encoding (TSHE) module constructs cross-modal hyperedges over the same word-level positions and intra-modal temporal-window hyperedges to jointly capture cross-modal co-occurrence and local temporal dependencies. Node-hyperedge-node propagation is further combined with instance dependent hyperedge routing to adaptively reweight hyperedge propagation according to sample-specific evidence. Finally, a Multi-task Fusion Output Layer (MFOL) aggregates structured representations across temporal and modality dimensions and jointly performs sentiment regression, polarity classification, and seven-class sentiment-level prediction. Experiments on CMU-MOSI, CMU-MOSEI, and their out-of-distribution splits demonstrate that SHF-MSA achieves competitive performance across both regression and classification tasks, with particularly strong results in strict polarity discrimination and distribution-shift evaluation.
Conventional EEG-based supervised emotion recognition typically requires large manually annotated multi-channel EEG data. The susceptibility of EEG to noise can lead to corruption of certain channels, making it challenging for emotion recognition from few available channels (denoted as few-channel in this paper). Masked Autoencoder (MAE) can learn highly generalizable representations by reconstructing the original signals from the visible subset of channels. However, most studies do not consider temporal masking, and random spatial masking overlooks the brain's inherent structure, impeding reconstruction efficacy. In this paper, we propose a novel pre-training framework, STMAE-Few, based on a spatial-temporal masked autoencoder, which introduces spatial-temporal masking strategies at both the EEG channel and frame levels to extract robust feature representations and enhance the performance of few-channel emotion recognition in real-world scenarios. Inspired by neuroscience, our approach integrates region and region-selection spatial masking strategies and introduces true position encoding during pre-training to capture richer spatial information. Extensive experimental results on SEED, SEED-IV, and SEED-V datasets demonstrate that our pre-training framework can effectively capture the temporal dynamics and spatial correlations within and between different brain regions.
