Stress and mental workload (MWL) frequently co-occur and share physiological signatures, reducing discriminability and limiting cross-subject generalisation. To address this, we implemented an immersive virtual reality (VR) adaptation of the Trier Social Stress Test (TSST) using a 2×2 factorial design that manipulated psychosocial stress (high/low) and MWL (high/low), enabling controlled evaluation under co-occurrence. We recorded electroencephalography (EEG), electrodermal activity, heart rate and heart rate variability, and pupillometry in 44 participants. Classification was used not as a methodological contribution or classifier benchmark, but as a probe of cross-subject physiological discriminability under co-occurring stress and MWL. Using leave-one-subject-out cross-validation, MWL was classified above chance across stress levels, whereas stress did not generalise beyond chance despite robust subjective effects. EEG spectral ratio features showed greater stability and stronger MWL discriminability than absolute power measures, while autonomic features showed limited state specificity. These findings indicate that physiological discriminability under co-occurring demands is state dependent. Under immersive, socially evaluative conditions, MWL exhibits modest but generalisable neural features across individuals, whereas stress appears substantially more sensitive to contextual and individual variation. This work clarifies the conditions under which cross-subject physiological generalisation is feasible and informs the evaluation and modelling of co-occurring cognitive-affective states in realistic settings.
Music has a profound power to induce and regulate emotional states. Inspired by the iso principle, in which listeners are guided towards a desired emotional state by first matching their current mood through music then gradually shifting the music towards the intended state, we explore the use of an AI-based automatic music generation (AI-AMG) system AffectMachine to support emotion regulation. We conducted a study with 54 participants and two versions of AffectMachine, Pop and Classical, to evaluate their ability to guide listeners towards calm and happy emotional states. Based on self-reported measures of emotion (arousal and valence), both versions of the AffectMachine successfully guided the majority of participants towards the target emotions, with stronger and more consistent effects on arousal than on valence. Interestingly, the high-arousal target emotion, Happy, tended to be induced more quickly than the low-arousal target emotion, Calm. We also found strong evidence of a positive correlation between change in arousal and change in heart rate. Our findings provide early evidence that AI AMG systems implementing the iso principle can support affect modulation, and offer insights for the design of adaptive music systems for therapeutic applications.
Multimodal large language models (MLLMs) have shown strong performance in generic visual understanding, yet their ability to perceive and reason about emotions in fine art remains underexplored. Unlike natural images, artworks frequently convey emotion indirectly through composition, color, style, and cultural context rather than explicit objects or events. To solve this, we proposed Art Emotion Reasoning Benchmark (AER-Bench), a benchmark for evaluating affective cognition over artworks with an explicit progression from low-level visual affect cues to high-level emotional reasoning. Specifically, we construct a structured artwork annotation schema and convert it into deterministic question templates spanning closed-ended perception, emotion ranking, dimensional affect estimation, open-ended description, and reasoning-oriented explanation. Using AER-Bench, we evaluate representative open-source and proprietary MLLMs with both objective metrics and rubric-based judging. Our results reveal a consistent gap between affective label prediction and evidence-grounded interpretation: Current MLLMs often produce plausible affective outputs, but struggle to ground them in perceptual, comparative, and contextual evidence. AER-Bench therefore provides a practical and extensible testbed for diagnosing and advancing emotionally aware visual intelligence.
Multimodal Emotion-Cause Pair Extraction (MECPE) aims to identify emotion utterances and their corresponding cause utterances in multi-party conversations. Although pre-trained models have delivered promising results, many existing methods are still developed primarily from an engineering perspective. Consequently, they tend to overlook the linguistic regularities that govern how emotion propagates in conversations. Emotion propagation in conversations is inherently unidirectional along the temporal axis, while its surface realization is further influenced by heterogeneity across modalities. We therefore characterize MECPE as facing two key challenges: 1) how to model the temporal dependencies of emotion propagation, and 2) how to capture both the complementary and inconsistent aspects of multimodal emotion features. To address these challenges, we propose a dynamic temporal differential multimodal emotion filtering model (DynTD). To encode the temporal propagation patterns of emotion in conversations, we introduce a Temporal Differential Table Fusion (TDTF) algorithm, which constructs an emotion cause table via a first-order temporal difference operator, explicitly modeling the unidirectional propagation and evolution of emotion over time. To account for the differences and complementarity in emotional expression across modalities, we further design a cross-modal Dynamic Gated Attention Filtering (DGF) mechanism. Using text as the primary modality, DGF employs cross-attention to extract complementary emotion related cues from the audio and video, while gated units suppress modality-induced discrepancies in emotional expression, thereby enabling adaptive fusion of multimodal emotion information. Experiments on several benchmark datasets demonstrate the superiority of DynTD, and extensive ablation studies confirm the effectiveness of conversational temporal modeling and multimodal discrepancy filtering for emotion analysis in conversations.
Understanding the psychological mechanisms underlying individual participation is essential for modeling information propagation dynamics on social media platforms. As a core psychological construct, personality significantly influences individuals' affective responses and behavioral decisions. However, existing propagation models often treat users as homogeneous actors and fail to sufficiently capture how personality traits shape users' affective participation processes, including commenting behavior, state transitions, and cross-platform propagation. To address this limitation, this study proposes the MP-SL3CI (Multi-platform Susceptible–Latent–3Commenting–Immune) model, which incorporates personality traits into a two-layer coupled network to examine dissemination on multi-layer platforms. Users are categorized into emotional, rational, and apathetic types to analyze personality effects on commenting behavior and cross-platform migration. Personality traits are extracted from user comments using pre-trained models, and simulations with real-world communication data are conducted to more realistically reproduce information dissemination processes across multi-layered networks. Numerical analyses, including comparative and ablation experiments, demonstrate that incorporating personality traits and multi-platform interaction mechanisms significantly improves the model's explanatory capacity and yields superior fitting performance compared with baseline models. Sensitivity analysis further identifies key factors influencing user decisions and cross-platform migration, providing theoretical support for propagation dynamics research and communication strategy design.
Music-related emotion recognition (MRER) aims to automatically predict emotional states based on different musical forms (e.g., lyrics, music, and music videos). Despite notable advancements in the field, MRER still faces the following challenges: 1) heterogeneity in emotion representations across datasets, including categorical and dimensional labels; 2) scarcity of comprehensive modalities and emotion annotations (e.g., static and dynamic) in existing music video emotion datasets; and 3) absence of a benchmark that contains various tasks and evaluation metrics for MRER. In this paper, we first propose a unified emotion representation consisting of emotion category and intensity, along with correlated conversion strategies to integrate disparate labels. Building upon the unified representation, we introduce an innovative emotion annotation framework MVAnno, which employs a hierarchical continual fine-tuning process on multiple modalities to obtain accurate emotion annotations. We also construct a large-scale music video emotion dataset MVEmo, which comprises 11 K samples, with 11 K static emotion labels and 5 M dynamic emotion labels. Finally, we present MVEmo-Bench, a multi-task benchmark with evaluation metrics specifically designed for MRER tasks and the natural language outputs generated by large language models (LLMs). Our work makes a significant contribution to MRER by addressing key challenges and providing robust foundations for future research.
Harmful memes are widely spread across the Internet, creating an urgent need for effective detection. Memes often embed cultural metaphors and social biases within concise visual–textual expressions. This high degree of semantic compression makes fine-grained harmful meme detection particularly challenging. Existing approaches either rely on large-scale annotated data or depend on large multimodal models for computationally expensive rationale generation, limiting scalability and generalization. To address these challenges, we propose DeMeme, a semantic decompression framework designed for lightweight adaptation and interpretability. In the first stage, the Risk-Aware Routing module efficiently estimates potential harmfulness using semantic anchors, routing high-risk memes to the retrieval stage for further interpretation. In the second stage, the Semantic Decompression Retrieval module retrieves high-similarity “Term–Definition” pairs, reconstructing hidden harmful semantics into explicit cultural and emotional cues. Experiments on three English and Chinese benchmarks show that the training-free Retrieve-All variant achieves strong zero-shot performance. These results confirm the effectiveness and cross-cultural adaptability of semantic decompression for fine-grained harmful meme understanding.
Affective art interaction provides a low-threshold way for users to externalize and regulate their emotions through creative activities. Current intelligent affective art interaction systems predominantly rely on text- or audio-based guidance, successfully assisting users in expressing their emotions. However, these systems often fall short in providing sufficient social companionship, visual sensing, and embodied feedback throughout the creative process. To address this, we present a 3D embodied agent-mediated affective art interaction system. Rather than treating the agent as a general embodied conversational assistant, the proposed system centers on an Affective-Art Composition Game, where users express and reshape affect through interaction and agent-guided art design. The system features two main components: an Affective-Art Composition Game and an affective conversational agent with a customizable 3D avatar providing socially present and embodied feedback. To address insufficient implicit affect capture, we introduce a semantic-guided video emotion perception module that contributes facial-video evidence to user-state understanding. To improve the avatar's behavioral feedback, we introduce a structure-aware avatar rigging module that supports stable embodied expression. On the MAFW benchmark, the proposed emotion-perception module achieves 44.45% UAR and 58.67% WAR, while the rigging module reduces joint-localization and skinning errors. User evaluations indicate that enhanced sensing improves perceived empathy and that improved rigging strengthens social presence and affinity. Collectively, this paper shows the potential of 3D embodied agent-mediated affective art interaction.
With the rise of social media, Multimodal Sentiment Analysis has gained growing attention. However, existing approaches still face three major challenges: (i) how to stabilize modality-specific representations and reduce the influence of inherent noise; (ii) how to enhance intra-modal discriminability so as to effectively preserve the heterogeneity of different modalities during fusion; and (iii) how to balance cross-modal semantic conflicts while fully leveraging the textual modality information. To address these issues, we propose a Semantic Purification and Time-Causal Stabilization with Energy-Inspired Arbitration Framework (SCEF). SCEF consists of three key modules: (i) the Semantic Causal Purification and Stabilization Module (SCPS) operates from both semantic and causal perspectives to effectively suppress inherent noise in textual representations, enhance their dynamic stability, and strengthen their intra-modal discriminability; (ii) the Causal Time-Frequency Heterogeneous Enhancement Module (CTHE) employs time-causal multi-scale time-frequency modeling to suppress the inherent noise of the visual and acoustic modalities while enhancing their structural characteristics and intra-modal discriminability; and (iii) the Energy-Inspired Recalibration and Arbitration Module (ERA) processes cross-modal conflicts through energy-driven recalibration and arbitration strategies, and implicitly strengthens the role of the textual modality via Stochastic Modality Dropout. Extensive cross-lingual experiments show that SCEF achieves superior performance across multiple benchmarks, reaching 90.85% Acc-2 on the CMU-MOSI dataset. The source code is publicly available at: https://github.com/CICC-XJTLU/SPTCS-EIA
Automated depression detection, which aims to assess the severity of a subject's depression, has witnessed rapid advancements in recent years yet faces persistent challenges. First, existing models predominantly focus on static facial features or short-term expressions, failing to adequately model psychomotor retardation, the phenomenon of delayed or slowed responses to stimuli. This cardinal clinical sign of depression involves inhibited temporal dynamics that are crucial for diagnosis. The omission of these dynamics critically limits the ability of current models to capture latent response patterns, such as reaction latency, that are essential for distinguishing clinical depression from transient sadness. Second, the generalization performance of current models is frequently suboptimal. Due to the limited scale of available datasets, these models tend to confound identity-specific or other non-depressive facial characteristics with key depressive indicators, thereby impairing their generalization capabilities. To address these limitations, we propose a novel Depression Feature Enhancement Module (DFEM). This module leverages a delay-aware proxy to capture slowed temporal dynamics while fusing long-term cues with short-term dynamics to enhance feature discriminability. Furthermore, to bolster generalization, we introduce an Identity-aware Prototypical Decoupling Regularized Module (IPDRM). This mechanism imposes a structured organization on identity-specific information within the feature space, implicitly compelling the network to encode depression-relevant cues in the residual feature variations, significantly enhancing model robustness. Extensive experiments on multiple benchmark datasets demonstrate that our proposed model achieves state-of-the-art performance and exhibits superior generalization capabilities. The code is publicly available at https://github.com/code-jiaqi/PRADD.
As a spontaneous non-intrusive biometric cue, gait conveys reliable affective information. However, the subtle and fine-grained motion of joints makes gait emotion recognition more challenging than general emotion recognition, particularly under the constraints of dataset scarcity and the high complexity of extracting discriminative cues from subtle motions. While existing models demonstrate competitive performance, they often do not account for the inherent heterogeneity of the dataset. Specifically, data from different acquisition devices exhibit clear cross-modality variations, which complicate the development of a unified emotion representation. In this work, we present CTR-Gait, a single-stream network designed for semantic-aware and fine-grained gait emotion recognition. The backbone integrates Channel-wise Topology Refined graph convolutions with Temporal Transformers to capture dynamic spatial correlations and global temporal dependencies. To structure the affective feature space, we introduce Emotion-Semantic Contrastive Learning (ESCL), which utilizes learnable Valence-Arousal centers to achieve semantically consistent emotion clustering. Furthermore, to tackle the inherent ambiguity in distinguishing subtle negative emotions, we design a specialized SadExpert module that dynamically refines predictions for the minority class. With 3.76 M parameters and 0.56 GFLOPs, CTR-Gait achieves 92.66% accuracy on the Emotion-Gait benchmark, surpassing existing state-of-the-art methods. At the same time, we conducted UAV evaluations of the field sequences with the model, and the results demonstrate the model's potential for practical deployment.
Micro-expressions (MEs) are spontaneous, subtle facial movements that reveal genuine emotions, making them indispensable for high-stakes applications like security and clinical diagnosis. However, Micro-Expression Recognition (MER) is currently hindered by a dual bottleneck: the intrinsic difficulty of extracting discriminative features from subtle motions and the extrinsic privacy constraints that restrict centralized training in real-world scenarios. To address these challenges, we propose FED-PsyAU, a unified framework that synergizes psychological insights with privacy-preserving computing. First, addressing the feature representation challenge, we propose a psychological study, which quantifies the coordination and incoordination between upper and lower facial Action Units (AUs). Leveraging these priors, we develop a Dynamic Prior Knowledge-based Graph Attention Network (DPK-GAT) combined with a novel Spatial Position-Aware Transformer (SPAT) and an Expression-Specific Attention (ESA) module. This architecture effectively integrates structural psychological constraints with data-driven statistical patterns to learn hierarchical facial motion features. Furthermore, to tackle data privacy and scarcity, we integrate this model into a Personalized Federated Learning (P-FedProx) framework, enabling collaborative training across distributed clients without raw data exchange. Extensive experiments on standard ME databases demonstrate that our method achieves state-of-the-art performance in both centralized and federated settings, validating the efficacy of incorporating psychological topology into privacy-preserving systems. Our implementation is publicly available at https://github.com/MELABIPCAS/FED-PsyAU.git.
Facial Action Units (FAUs) capture the micro-level facial movement patterns of individuals during emotional expression, emotional response, and social interaction processes. While effectively preventing the leakage of personal identity information, they provide a robust representation for depression recognition with the added advantage of privacy protection. However, existing studies lack the guidance of external emotional stimuli and fail to adequately model interpretability. To address this, we have constructed the Emotional Image Viewing Experiment (EIVE) dataset for depression recognition based on FAUs. Meanwhile, We present SFENet, a shapelet-based facial expression interpretability model for depression recognition. Specifically, we extract shapelets for each FAU from subject samples using perceptually important points and construct shapelet-based initialization convolution kernels. Building upon this kernel, we propose a shapelet-convolution-based historical information enhancement module that strengthens feature extraction through historical information aggregation and time-scale interaction. Additionally, we design a sliding-window-based shapelet-sample similarity loss to update shapelet features, optimizing model training by measuring the alignment between features and sample sequences. Interpretability is derived from the parameters initialized by the shapelet convolution kernel and fitted through model training. SFENet achieves strong competitive performance in both EIVE and AVEC 2014, revealing that the case group exhibits a stronger tendency for negative facial expression unit activation under different emotional conditions. Our work is available at https://github.com/yjy-97/SFENet.
In scenarios with missing modalities, although existing multimodal sentiment analysis has made significant progress in handling incomplete modalities, there remains room for improvement in modality feature learning and cross-modal alignment. To better bridge the semantic gap between different modalities and reduce reconstruction errors for missing modalities, A multimodal sentiment analysis model based on information enhancement toward modal absence, referred to as IETMB, was proposed. The model is designed to fully capture modality-invariant features through a six-branch cross-modal Transformer and contrastive learning, while encoding modality-specific features to reduce modality distribution gaps and fully utilize multimodal information. Then, a cascaded attention self-encoder-based feature reconstruction module is designed to generate robust reconstructed features by enhancing the feature representation step-by-step with multi-level attention self-encoders. Finally, the residual information enhancement module introduces a residual component in order to enhance the semantic information of the reconstructed features to generate semantically rich fusion features. Experimental results on the public datasets IEMOCAP, MSP-IMPROV, and CMU-MOSI show that the model improves in performance compared with the advanced models. The weighted accuracy (WA) improved by 0.57% and 0.33% in IEMOCAP and MSP-IMPROV, respectively, and the mean absolute error (MAE) decreased by 0.0069 on CMU-MOSI.
Reliable recognition of human emotions from multimodal signals is important for human-computer interaction and affective intelligence systems. However, multimodal sentiment analysis remains challenged by heterogeneous modality representations, insufficient modeling of high-order interactions, and limited generalization under distribution shifts. To address these issues, this paper proposes SHF-MSA, a hypergraph-based multimodal sentiment analysis model that integrates representation coordination, structured high-order fusion, and multi-task learning. First, a Unified Pre-encoding Layer (UPL) employs modality-specific encoding, shared-space projection, and statistical normalization to reduce representation discrepancies among textual, acoustic, and visual modalities while preserving modality-specific and temporal information. Next, a Temporal Synergistic Hypergraph Encoding (TSHE) module constructs cross-modal hyperedges over the same word-level positions and intra-modal temporal-window hyperedges to jointly capture cross-modal co-occurrence and local temporal dependencies. Node-hyperedge-node propagation is further combined with instance dependent hyperedge routing to adaptively reweight hyperedge propagation according to sample-specific evidence. Finally, a Multi-task Fusion Output Layer (MFOL) aggregates structured representations across temporal and modality dimensions and jointly performs sentiment regression, polarity classification, and seven-class sentiment-level prediction. Experiments on CMU-MOSI, CMU-MOSEI, and their out-of-distribution splits demonstrate that SHF-MSA achieves competitive performance across both regression and classification tasks, with particularly strong results in strict polarity discrimination and distribution-shift evaluation.
Conventional EEG-based supervised emotion recognition typically requires large manually annotated multi-channel EEG data. The susceptibility of EEG to noise can lead to corruption of certain channels, making it challenging for emotion recognition from few available channels (denoted as few-channel in this paper). Masked Autoencoder (MAE) can learn highly generalizable representations by reconstructing the original signals from the visible subset of channels. However, most studies do not consider temporal masking, and random spatial masking overlooks the brain's inherent structure, impeding reconstruction efficacy. In this paper, we propose a novel pre-training framework, STMAE-Few, based on a spatial-temporal masked autoencoder, which introduces spatial-temporal masking strategies at both the EEG channel and frame levels to extract robust feature representations and enhance the performance of few-channel emotion recognition in real-world scenarios. Inspired by neuroscience, our approach integrates region and region-selection spatial masking strategies and introduces true position encoding during pre-training to capture richer spatial information. Extensive experimental results on SEED, SEED-IV, and SEED-V datasets demonstrate that our pre-training framework can effectively capture the temporal dynamics and spatial correlations within and between different brain regions.
Affect sensing during natural social interaction remains a bottleneck for scalable digital mental health, as physiology-based Emotion Recognition in Conversation (ERC) is often validated in scripted protocols or human–human interactions with high interpersonal variability, and mostly in healthy cohorts. We present a controllable, semi-immersive virtual reality (VR) paradigm based on open-ended conversations with large-language-model-driven Virtual Humans whose affective stance is experimentally manipulated. Multimodal signals were recorded from 98 participants (58 controls, 40 with depressive symptoms) using EEG, ECG/HRV, EDA, and eye-tracking. Subject-independent XGBoost models were evaluated with nested cross-validation under two labeling families: theoretical (design-based) and participant-based labels. Theoretical labels consistently outperformed participant-based supervision. For four-class recognition, the multimodal late-fusion model achieved 0.83 Macro-F1. For five-class emotion recognition, EEG and the late-fusion multimodal learning achieved the same best performance (0.84 accuracy, 0.83 Macro-F1). Training on the combined cohort improved generalization across subgroups (EEG Macro-F1=0.82 in controls; 0.86 in the depressive-symptom group). SHAP analyses revealed stable fronto-central-parietal EEG spectral patterns across groups, with depressive symptoms mainly modulating effect magnitude. Overall, Virtual Humans provide a scalable framework for reproducible conversational affect elicitation and physiology-based decoding of interaction-induced affective responses in mental-health-relevant populations.
Visual emotion inference aims to predict human-perceived affective states from images, bridging low-level visual features with high-level emotional semantics. However, existing methods often rely on direct mappings from global features to emotion labels, lacking a mechanism to reflect the cognitive process by which humans perceive and interpret emotions. To address this limitation, we propose a dual-process visual emotion reasoning framework that integrates Multi-Patch Attention Flow (MPAF) and Fuzzy Emotional Semantic Modeling (FESM). MPAF is designed as the perceptual pathway of the proposed emotion reasoning framework, simulating how humans distribute attention and integrate emotional cues across patches. The image is divided into overlapping patches through which an attention flow dynamically integrates emotional cues, approximating how human vision distributes and refines attention across a scene. Meanwhile, FESM serves as the cognitive pathway of our frame-work based on fuzzy mathematics, projecting each patch into a continuous affective semantic space. Instead of fixed emotion labels, it assigns graded membership values across multiple fuzzy emotion concepts, formally simulating the ambiguity, continuity, and polysemy of human visual emotion reasoning. Within this unified framework, Patches of Interest (PoIs) naturally emerge as regions most aligned with the global affective state, providing interpretable insight into how local cues anchor overall emotional appraisal. Experimental results demonstrate that the proposed framework not only achieves effective emotion prediction but also offers cognitively meaningful interpretability, providing a promising step toward cognitively inspired visual emotion reasoning.
Continuous affect estimation has gained significant momentum in recent years, with promising applications in mental health assessment, human-computer interaction, and multimedia content analysis. Researchers have developed various deep learning approaches leveraging audio-visual cues to predict continuous valence-arousal values from videos. Despite substantial progress, current methods still suffer from two critical limitations: they typically adopt deterministic mappings that fail to capture the inherent ambiguity of emotions, and they often overlook psychological constraints between valence and arousal dimensions, potentially yielding implausible emotional representations. To address these challenges, we propose DiffAffect, the first framework that leverages conditional diffusion models for continuous affect estimation. Our approach reformulates affect estimation as a denoising diffusion process, employing a multimodal feature encoder with cross-modal attention and a conditional diffusion decoder to transform noise into valence-arousal predictions progressively. Furthermore, we introduce an asymmetric relation-guided sampling strategy that integrates psychological correlations between dimensions into the generation process. We evaluated DiffAffect on two benchmark datasets, achieving state-of-the-art results on the challenging Aff-Wild2 dataset. Our approach also demonstrates consistent performance on the AFEW-VA dataset, confirming its robustness across both in-the-wild and controlled scenarios. These results validate that generative diffusion models effectively capture the subjective and uncertain nature of emotional expressions, making them particularly suitable for real-world affective computing applications.
Cross-subject EEG emotion recognition faces significant challenges due to the substantial variability in brain activity patterns across individuals and the non-stationary nature of EEG signals. Existing methods based on Euclidean space or conventional graph neural networks (GNNs) struggle to effectively capture the brain's functional connectivity’s hierarchical and non-Euclidean structure. Moreover, subject-specific information often becomes entangled with emotion-related features, degrading the model's generalization ability to unseen individuals. We propose a Hyperbolic Transformer framework with identity-invariant representation to address these issues. The framework first employs a GraphSAGE-based local graph encoder to extract region-level EEG representations. Then it models global hierarchical dependencies in Lorentzian geometry via a hyperbolic Transformer. In addition, we design an identity-invariance learning module to explicitly decouple identity-related and emotion-related representations, and incorporate a gradient reversal strategy to suppress identity information leakage. Furthermore, we introduce a hyperbolic contrastive loss to enhance inter-class separability while preserving manifold geometric consistency. Experimental results on the SEED, SEED IV, MPED, and large-scale FACED datasets demonstrate that the proposed method significantly outperforms state-of-the-art approaches in cross-subject emotion recognition, especially in scenarios with significant inter-subject variability.
