This study addresses a critical challenge in Virtual Reality (VR): the detection of visually induced motion sickness (VIMS), a factor that significantly limits prolonged VR use. By integrating head, eye, and mouth movement data, we propose a novel multi-modality-based approach for early detection of VIMS. Our Attention-VR model is designed to dynamically assign weights to the more important modalities among the three (head, eye, and mouth) for identifying the early signs of VIMS. This paper reports on two studies. In Study 1, we developed our predictive models by analyzing multi-modal data from 38 participants experiencing a customized VR roller coaster environment. In Study 2, we validated these models using data from commercial VR games. By integrating diverse physical movement signals, Attention-VR significantly improves the precision and interpretability of VIMS predictions, providing new insights into the relative importance of each modality. We discuss the limitations of the current approach and suggest directions for future research.

Empathy modeling in online interactions presents a significant challenge due to its dynamic, context-sensitive, and bidirectional nature. To address these complexities, we propose the Bi-directional MLP-Mixer (Bi-Mixer) and dynamic weights fusion model—a novel neural architecture that captures temporal dependencies in both forward and reverse directions and performs dynamic multi-modal fusion using a cross-attention mechanism. The model adaptively reweights input from visual, audio, text, and biological modalities based on contextual cues, enabling more accurate real-time empathy prediction. To support the training and evaluation of our proposed model, we build the Multi-modal online interaction EMPathy (Multi-EMP) dataset, consisting of unscripted dyadic conversations recorded through online video conferencing. The dataset includes four synchronized modalities: video, audio, text, and bio-signals such as electrodermal activity, blood volume pulse, temperature, and metabolic equivalent of task. It enables dual empathy assessment based on both explicit self-reports and computed emotional alignment between speaker and listener. Our contributions include: (1) the Bi-Mixer and dynamic weights fusion model for bidirectional and adaptive multi-modal representation learning, (2) the release of a comprehensive multi-modal dataset for naturalistic empathy research, and (3) a dual-perspective framework for empathy evaluation.

Early detection of depression is crucial because depression can lead to suicide if the symptoms are left unrecognized or untreated. In hospitals, self-administered questionnaires and interviews are employed to diagnose depression. Although doctors spend considerable time interviewing patients to understand their conditions, depression is a heterogeneous syndrome that makes accurate diagnosis challenging. Therefore, the biological aspects of depression must be investigated to address the limitations of traditional diagnostic methods. Audio data can be easily collected in daily life. Hence, we propose a multimodal fusion cross-modality model that applies audio and text to detect depression. The proposed model achieved F1-scores of 0.67, 0.81, and 0.61 on the Distress Analysis Interview Corpus, Emotional Audio and Textual Depression Corpus, and Korean Depression datasets. The model is designed to be lightweight, reducing the number of parameters while maintaining model accuracy with fewer parameters so that it can be employed in pervasive devices. We used English, Chinese, and Korean depression datasets to evaluate the performance of the proposed model across languages. The cross-language experiments confirm that the proposed model can be applied in other languages, even if the model is not trained in the same vocabulary. This finding suggests that the model has learned distinctive depression characteristics by combining nonlinguistic speech features and linguistic textual features. Therefore, this research is expected to detect depression in everyday life across languages and devices.

We propose a novel multi-modalities based approach that predicts visually induced motion sickness (VIMS) by integrating head, eye, and mouth movement data. Our Attention-VR model dynamically assigns higher weights to the most informative of the three modalities for identifying early signs of VIMS. Multi-modal data from 38 participants in a customized VR rollercoaster environment were collected to develop our predictive model. By combining diverse physical movement signals, Attention-VR significantly improves prediction accuracy and interpretability, clarifying the relative importance of each modality.

Emotion recognition in video aims to estimate human emotions using acoustic, visual, and linguistic information. This problem is considered multimodal and requires learning different modalities, such as visual, verbal, and vocal cues. Although previous studies have focused on developing sophisticated deep learning models, this work proposes a different approach using dynamic restrained adaptive loss inspired by multitask learning to understand multimodal inputs jointly. This training strategy allows predictions from one modality to enhance the accuracy of predictions from other modalities, mirroring the concept of multitask learning, where the results of one task can improve the performance of related tasks. Furthermore, this work introduces the extended multimodal bottleneck transformer, an efficient and effective mid-fusion method designed for problems involving more than two modalities to enhance the performance of emotion recognition systems. The proposed method significantly improves results compared to other end-to-end multimodal fusion techniques on three multimodal benchmarks—Interactive Emotional Dyadic Motion Capture (IEMOCAP), Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI), and the Chinese Multimodal Sentiment Analysis dataset with independent unimodal annotations (CH-SIMS).

Social perception is a crucial psychological concept that explains how we understand and interpret others and their behaviors. It encompasses the complex process of discerning individual characteristics, intentions, and emotions, significantly influencing social interactions and decision-making. In this paper, we propose a modality weights based fusion model for predicting 16 social attributes in the MuSe 2024 Challenge subtask MuSe-Perception. The proposed model utilizes visual, audio, and text data from the Chief Executive Officers' (CEOs') interviews to predict these 16 social attributes. This study mainly focuses on audio feature learning and the cross-attention of audio model outputs. Furthermore, we develop each modality weight-based fusion network that combines the outputs of each modality according to their weights. Experimental results show that the proposed model achieved competitive performance for some social attributes, but there were limitations to attain consistent overall performance. Based on these results, future work includes collecting more extended CEO video data and learning the importance of each modality for different attributes. This study is expected to contribute to developing a system for analyzing investment potential by predicting the CEOs' social attributes.

We present the Group Cohesion and Emotion (GCE) dataset, which comprises 1029 segmented films sourced from YouTube. These videos encompass a range of interactions, including interviews, meetings, informal discussions, and other similar contexts. In the annotation process, graduate psychology students were tasked with assigning coherence levels, ranging from 1 to 7, and affective states as negative, neutral, or positive for each 30 s film. We introduce a foundational model that utilizes advanced visual and audio embedding techniques to investigate the concepts of group cohesion and group emotion prediction. The application of Multi-Head Attention (MHA) fusion is utilized to enhance the process of cross-representation learning. The scope of our investigation includes both unimodal and multimodal techniques, which provide insights into the prediction of group cohesion and the detection of group emotion. The results emphasize the effectiveness of the GCE dataset in examining the level of group unity and emotional conditions.

Physiological signals are widely used in the recognition of affective status. Recording of such physiological signals involves elicitation of emotions through different stimuli including video-based stimulus. Considering that the same stimulus videos often induce different emotions in different individuals, emotion recognition in such a scenario requires consideration of the individual differences in the consumption of the stimulus content. With this as our goal, we present a Physiological dataset for Multimodal Emotion Recognition (PhyMER) for studying emotion through physiological response with personality as a context. The PhyMER dataset consists of electroencephalogram (EEG), electrodermal activity (EDA), blood volume pulse (BVP), and skin temperature along with the personality traits of 30 participants. We collected the video-based stimulus dataset for emotion elicitation and developed a web-based annotation tool for labeling felt emotions. We compared the stimulus labels and the self-annotation of felt emotions labeled during physiological data recording. Correlation among personalities was analyzed to study the impact of personality on the intensity of emotions in arousal and valence dimensions. Finally, we proposed a baseline model for the classification of emotions using physiological signals. The dataset is publicly available to the academic community for analysis of affective states and the development of emotion recognition models.

This paper presents a framework for automatically building a product data repository, overcoming the limitations of machine understanding and the time-consuming, costly nature of manual human annotation. In this study, we focus on Personal Protective Equipment (PPE), specifically respirator mask product categories, as a case study. First, we extract product specifications from the details and sub-details sections of product pages and build < attribute, attribute-values > metadata in dictionary form. Second, to process the data, we represent numerical data with a special token [NUM] and use hierarchical clustering based on Term Frequency-Inverse Document Frequency (TF-IDF) and cosine similarity for categorical data. Third, we propose an algorithm-based method to annotate product attributes and attribute values in product descriptions using the metadata, while minimizing human efforts. Additionally, we employ BIO tagging to extract location information for attribute values, streamlining the overall annotation process. This paper contributes to the field by presenting a method for automatically generating product-related datasets through an algorithm that enables the construction of metadata without any human intervention. This approach is a crucial step toward implementing a knowledge repository in a Cloud Manufacturing (CM) environment. Particularly, by addressing time and effort concerns in the dataset construction process, this method can significantly contribute to the creation of large-scale datasets.

Understanding emotions in conversation is a challenging task, as the sentences often have an implied meaning that is not generally understood in isolation. Efficient use of contextual information is essential for emotion recognition in conversations. Many published datasets provide contextual information for situations such as text-based online messaging, chatbots, and movie dialogues. However, such dialogue-based datasets are collected by selecting ideal conversational situations and thus do not include many variations in dialogue length and number of participants. Therefore, such datasets may not be applicable for emotion recognition in text-based movie transcripts, where scenes contain variations in the number of speakers and length of spoken sentences. We present a conversation dataset based on the Korean television show transcripts to analyze the emotions in presence of scene context. The Korean Drama Scene Transcript dataset for Emotion Recognition (KD-EmoR) is a text-based conversation dataset. We analyze three classes of complex emotions: euphoria, dysphoria, and neutral, in the scenes of a television drama to build a publicly available dataset for further research. We developed a context-aware deep learning model to classify emotions using the speaker-level context and scene context and achieved an F1-score of 0.63 on the proposed dataset.

Facial Action Coding System is an approach for modeling the complexity of human emotional expression. Automatic action unit (AU) detection is a crucial research area in human-computer interaction. This paper describes our submission to the third Affective Behavior Analysis in-the-wild (ABAW) competition 2022. We proposed a method for detecting facial action units in the video. In the first stage, a lightweight CNN-based feature extractor is employed to extract the feature map from each video frame. Then, an attention module is applied to refine the attention map. The attention encoded vector is derived using a weighted sum of the feature map and the attention scores later. Finally, the sigmoid function is used at the output layer to make the prediction suitable for multi-label AUs detection. We achieved a macro F1 score of 0.48 on the validation set and 0.4206 on the test set compared to 0.39 and 0.3650 from the ABAW challenge baseline model.

Facial Action Coding System is an approach for modeling the complexity of human emotional expression. Automatic action unit (AU) detection is a crucial research area in human-computer interaction. This paper describes our submission to the third Affective Behavior Analysis in-the-wild (ABAW) competition 2022. We proposed a method for detecting facial action units in the video. At the first stage, a lightweight CNN-based feature extractor is employed to extract the feature map from each video frame. Then, an attention module is applied to refine the attention map. The attention encoded vector is derived using a weighted sum of the feature map and the attention scores later. Finally, the sigmoid function is used at the output layer to make the prediction suitable for multi-label AUs detection. We achieved a macro F1 score of 0.48 on the ABAW challenge validation set compared to 0.39 from the baseline model.