Sign Language Generation (SLG) has received increasing attention in recent years, with various models aiming to produce natural and temporally coherent sign gestures from spoken or written language. However, SLG remains a challenging task due to its inherently one-to-many nature, where a single sentence can correspond to multiple valid gesture sequences, and the requirement for smooth, synchronized motion across multiple articulators. Although diffusion-based models capture diversity, their stochastic denoising often introduces temporal misalignment and motion artifacts. In this work, we propose SignFlow, a novel architecture for SLG based on conditional flow matching with optimal transport. By modeling deterministic flow paths guided by optimal transport and supervised via velocity fields, SignFlow generates gestures that are both semantically coherent and visually smooth. Experiments on the CSL-Daily dataset demonstrate that SignFlow achieves superior BLEU scores and DTW-based motion accuracy compared to both diffusion and autoregressive baselines.
A framework named perceptual functional spectrum analysis (pFSA) for analyzing how people perceive the multifunctional nonverbal behaviors that emerge in conversations is proposed. The goal is to elucidate the intrinsic nonverbal properties, called functional multiplicity and interpretational ambiguity, in a separable way. The former property is that a single behavior could imply multiple meanings, and the latter is that different observers could interpret the same behaviors differently. In the pFSA framework, the labels of multiple raters across multiple functions over time are represented as a third-order tensor. This study then formulated a semiorthogonal nonnegative tensor factorization (SO-NTF) that approximates the input tensor as a linear combination of the functional basis matrix, perceptual basis matrices, and perceptual coefficient matrices. The functional basis matrix consists of functional spectra that represent fundamental functionalities in conversations. The perceptual basis matrices represent the perceptual tendencies, which consist of the sensitivities of the raters to the fundamental functionalities. The perceptual coefficient matrices represent the temporal activations of the perceptual tendencies. The pFSA framework constructs the perceptual basis matrices to characterize both label reliability and diversity. This study targeted 32 head movement functions labeled by ten raters. The experimental results confirmed that pFSA could successfully analyze the levels of ambiguity for multiple functionalities, such as low ambiguity for addressing and backchannel functions and high ambiguity for thinking functions.
Disinformation poses serious challenges as it involves deliberate falsehoods that mislead the public, erode trust, and hinder informed decision-making, issues exacerbated by AI technologies like ChatGPT, which facilitate realistic generation while complicating detection. Existing methods often rely on large-scale multimodal datasets like NewsCLIPpings, but their limited language coverage and reliance on naive, easily detectable examples from social media or synthetic sources reduce their effectiveness, especially on out-of-domain (unseen) data. To address these limitations, we develop a novel multilingual, multimodal dataset curated from fact-checking websites, presenting a greater challenge for existing detection models. Notably, we find that multilingual training not only facilitates cross-lingual generalization but also improves robustness on monolingual data, highlighting the value of linguistic diversity in building generalizable OOC detectors. Our dataset uniquely incorporates supporting information, and we propose a simple training strategy that utilizes supportive information to enhance both learning efficiency (effective use of small-scale data) and detection accuracy, even on unseen data. The dataset and code will be publicly released.
Identifying affective responses of users engaged with digital information can provide valuable information on user experience and can be used for user modelling, retrieval, and content recommendation. Methods for recognising affective states often rely on supervised learning from a single modality, such as decoding affective information from content via computer vision, human physiology, or brain responses. These approaches assume the availability of paired data that contain the modality from which the prediction is made and the corresponding affective state label that is used to supervise the model. Unlike previous research, we introduce an approach to decode affective states via bimodal contrastive learning without using any externally provided affective state labels. Our method, entitled BALE, uses only bimodal data (paired brain recordings and images) to self-learn latent representations by contrasting image representations and the representations of the brain responses evoked when humans perceive the visual stimuli. We demonstrate the effectiveness of our approach using two publicly available datasets: fNIRS recordings of images showing various levels of valence (positivity/negativity) and fMRI recordings of individuals viewing human faces showing different emotions (joy/anger/contempt/pride/neutral). We evaluate our model on two tasks: classification of affective states and ranking of visual stimuli based on the affective responses they evoke when perceived. We report the first successful results for bimodal learning of affective states without labels and report performance approaching models trained with labelled supervision data. The results provide evidence that subjective affective states can be captured from multimodal interactions. Our source code is openly released at https://github.com/VadymV/BALE/.
Accurate pain assessment remains a major challenge in clinical practice, particularly in non-communicative patients where self-report is unfeasible. This study presents a hybrid, explainable approach for automatic pain classification from physiological signals, developed in the context of the AI4PAIN 2025 Challenge. We address key limitations of the dataset (limited size, signal discontinuity, and quality degradation) by combining deep learning (DL) for pattern discovery with classical machine learning (ML) for robust classification. In Stage 1, a CNN-LSTM model with attention and explainability modules, Gradient-weighted Class Activation Maps (GradCAM) and feature ablation, identifies temporally and modality relevant signal patterns. Guided by these insights, Stage 2 extracts a compact set of physiologically grounded features from photoplethysmography (PPG), electrodermal activity (EDA), and chest respiratory effort (RESP) signals. Stage 3 employs an ensemble of optimized ML models (CatBoost, XGBoost, LightGBM) trained on these features, achieving 0.84 F1-macro on validation data. Explainability analysis confirms that the model captures clinically relevant pain markers, such as sympathetic arousal (EDA), disrupted pulse variability, in terms of amplitude and width, and altered respiratory patterns. Despite a validation-to-test performance drop (0.14), our XAI-guided methodology demonstrates strong interpretability, highlighting its potential for trustworthy pain assessment in real-world clinical settings.
Human interaction with the environment is inherently multimodal, with cues from speech, sound, touch, and body language naturally reinforcing and complementing each other [1]. Thus, achieving natural human-AI interaction requires methods that can integrate and coordinate multiple modalities effectively. This tutorial introduces participants to building multimodal human-AI interaction systems using agentic AI patterns, using emerging frameworks such as LangFlow, which provides an intuitive interface for designing and debugging agent workflows; LangGraph, a scalable graph-based framework for orchestrating multi-agent systems; and the Model Context Protocol (MCP), which standardises how agents access shared context and tools. Participants will learn to orchestrate AI agents capable of interpreting speech, analysing images, and adapting dynamically to user intent, history, and context. Combining conceptual grounding with hands-on implementation, the tutorial equips researchers with the skills to prototype and explore multimodal interaction paradigms, contributing to the design of natural and engaging human-AI interfaces.
This workshop aims to advance research on cross-cultural multimodal interaction by establishing a platform for sharing methodologies, data, and insights related to nonverbal behavior across different linguistic and cultural contexts. We focus on the challenges of collecting and annotating multimodal data in a consistent manner across diverse locations, and on developing analysis methods that leverage recent advances in machine learning and large language models. By bringing together researchers from various disciplines and regions, the workshop seeks to foster international collaboration, promote the harmonization of annotation standards, and encourage interdisciplinary approaches. The outcomes are expected to contribute to a deeper understanding of cultural differences in nonverbal communication and to the development of technologies that enable natural and effective human-human and human-machine interaction in multicultural societies.
Pain recognition using physiological signals is a critical step toward advancing automatic pain assessment, particularly in clinical and experimental settings. In this study, we present a machine learning-based approach to address the AI4PAIN 2025 challenge, which involves classifying pain intensity into three levels—No Pain, Low Pain, and High Pain—based on multimodal time-series signals including electrodermal activity (EDA), blood volume pulse (BVP), respiration (RESP), and peripheral oxygen saturation (SpO2). After extracting relevant statistical and frequency-domain features from each modality, we evaluated a series of classical machine learning models: RandomForestClassifier, LGBMClassifier, HistGradientBoostingClassifier, XGBClassifier, and CatBoostClassifier. Among these, the Random Forest Classifier emerged as the most robust model, achieving the highest test accuracy of 62.07%, indicating better generalization compared to others. Although the XGBClassifier attained the highest validation accuracy of 68%, its performance degraded more noticeably on the test set, suggesting potential overfitting. Both LGBMClassifier and HistGradientBoostingClassifier reached similar validation accuracies (around 64%), with the latter showing better class-wise balance under label imbalance conditions. However, Random Forest demonstrated the most consistent performance across both validation and test evaluations, making it the most reliable model in this study. The entire preprocessing, feature engineering, and classification pipeline developed for this task is encapsulated in our proposed system, Painthenticate, designed to support reproducible and interpretable pain classification. These findings highlight the value of model selection not only based on aggregate accuracy but also on consistency across class distributions in physiological pain classification tasks.
As generative vision models become increasingly integrated into downstream applications, there are increasing concerns about the cultural sensitivity and representational fairness of these models. As existing training datasets may be skewed towards certain dominant cultures and overlook the nuanced cultural contexts embedded in the data, the generative visual outputs often misrepresent or marginalize minority cultures. The cultural bias can be especially problematic in educational settings, where representation plays a critical role in shaping learners’ understanding and sense of identity. In this paper, we benchmark visual generative models through cultural lens, using an educational use case set in Singapore’s multi-cultural context. We introduce and utilize a Singapore-centric multi-cultural image dataset that reflects rich ethnic, religious, and social diversity. Using this dataset, we evaluate the performance of nine state-of-the-art generative models, including DALL-E 3 and SD3.5, on their ability to contextually represent multi-cultural diversity. Our findings reveal that even the top-performing models SD3, DALL-E 3, and Playground v2, with strong performance in fidelity, semantic alignment, and visual quality show variability across cultural subsets, with poorer performance with underrepresented or more nuanced cultural contexts. This case study contributes to a broader understanding of cultural representation and localization of visual generative technologies.
Interactive visual storytelling through conversational agents offers a means to enhance early childhood language learning. We developed a picture-guided conversational chatbot, driven by dense image captioning, for early childhood mother tongue language learning. However, state-of-the-art image captioning systems fall short in meeting the educational needs of young learners. They often lack cultural contextualization, use vocabulary that exceeds children’s developmental level, and fail to align with curriculum-relevant learning goals. We investigated a contextualized dense image captioning framework, which augments dense image captioning with cultural and curriculum-aligned keyword retrieval through a Retrieval-Augmented Generation (RAG) module. This enables the generation of culturally appropriate, age-level suitable, and educationally anchored captions that enhance learner engagement and pedagogical relevance. We demonstrate that our approach outperforms existing captioning models in terms of linguistic appropriateness, and curriculum and cultural alignment. The contextualized dense image captioning framework supports the development of culturally grounded, education-oriented conversational agents for young learners.
Training multimodal large language models (LLMs) for safety-critical assistive applications, especially those handling sensitive user data, presents challenges related to responsible deployment, privacy, and computational efficiency. Centralized training risks user privacy during fine-tuning, while resource-heavy methods limit deployment in real-world assistive scenarios. To advance safe and responsible multimodal interaction, we propose a federated learning approach that enhances user privacy through decentralized data processing, enabling model fine-tuning without compromising user data security. Additionally, we extend visual instruction tuning by applying efficient fine-tuning techniques to multimodal language-image instruction-following data. This process results in capable multimodal LLMs optimized for computational efficiency, trainable in similar to 36 hours on a single 2xA100 node, making it more accessible for inclusive deployment. We demonstrate a practical application of these advances through an assistive system designed for visually impaired users. This application utilizes a privacy-preserving and efficient multimodal LLM to provide real-time, interactive, and descriptive engagement with the environment, enhancing user understanding, autonomy, and safety. Our results show that combining federated learning with efficient visual instruction enables secure, transparent, and scalable multimodal LLMs. These advances support responsible AI deployment in assistive technologies, promoting accessibility, trust, and ethical impact for individuals with disabilities and the elderly, while upholding ethical integrity and societal benefit.
An analytic framework named an interactional functional spectrum analysis (iFSA) is proposed to reveal how people interact with each other via multimodal nonverbal behaviors in multiparty conversations, focusing on their interactional functional aspects. Based on the representation called functional spectrum, which is the distribution of perceptual intensities over multiple functions of nonverbal behaviors, this study extends such approach to analyze multiparty multimodal multifunctional interactions. More specifically, the iFSA introduces three key extensions: i) nonverbal modalities consisting of facial expressions, head movements, and gaze behaviors; ii) group-level interactions consisting of a speaker, addressee, and other listeners; and iii) temporal spectrum pooling to account for reaction time. From the multiparty multimodal functional spectra, the iFSA conducts spectrum decomposition via semiorthogonal nonnegative matrix factorization (SO-NMF), which approximates the input spectra as the product of a basis matrix called interactional functional basis and a coefficient matrix called interactional functional spectrum. The former represents fundamental patterns of multimodal interactions, and the latter indicates the temporal activation of each basis vector, i.e., each interaction pattern. The experiments targeting four-party conversation data revealed several essential interactions, such as the speaker’s full-modal addressing response with attentive listening by the addressee and other listeners.
Persuasion is to change or influence a person's attitude or behavior without coercion or deception. Combined with personalization, personalized persuasion has demonstrated greater impact on influence. From persuasion theories in social psychology, emotion, cognition and personality are considered important factors. These factors are also under active research for automated detection within the computing community. How mental state and personality play a role in the process of persuasion and in persuasion outcomes has not been investigated. In this paper, we investigated the difference of mental state and personality between persuaders and receivers in a dyadic conversation where different persuasion outcome was produced. A video dataset of 24 participants in pairs debating in a survival task was collected. Arousal, valence, gaze direction, speech length and Big Five Inventory were annotated and processed to represent emotion, attention, mental load and personality during each dialog. Statistical analysis results show that persuaders’ arousal was significantly higher than receivers’ to disagreement outcome, persuaders imposed lower mental load on receivers to agreement outcome, and when persuaders’ extraversion score was higher than receivers’, the agreement response rate was higher. This research contributes to the understanding of mental state and personality of both persuaders and receivers in the context of dyadic interaction for a potential of dynamic and personalized adaptation.
We present DriveWise, a personalized driver coaching system that integrates naturalistic performance assessment, CARLA-simulated lesson generation, and eye-tracking-based risk perception feedback. Driving skills are evaluated through unscripted simulation, where collisions and near-misses are detected to quantify risk. Based on these incidents, tailored interactive lessons are generated to address individual weaknesses. Eye-tracking data further identifes gaps in visual attention, enabling cognitive-level coaching. DriveWise offers a scalable approach to enhancing both behavioral and perceptual aspects of driving safety.
Objective and automated pain intensity assessment is a critical challenge in healthcare, with physiological signals offering a promising, non-invasive avenue for quantification. This paper addresses the AI4Pain Grand Challenge task of classifying pain into three levels (No Pain, Low Pain, High Pain) using multimodal physiological data, including Electrodermal Activity (EDA), Blood Volume Pulse (BVP), Respiration (RESP), and SpO2. We conduct a systematic comparison of two distinct modeling paradigms: (1) a feature-driven approach using TabPFN, a pre-trained tabular learner, which leverages extensive handcrafted features, ensembling, and stacking techniques; and (2) a hybrid deep learning model, Transformer with Features (TwF), which combines a Transformer-based encoder for raw signal analysis with a curated set of physiological features. Our experiments demonstrate that the feature-driven TabPFN approach significantly outperforms the deep temporal model. The final stacked TabPFN model achieved a test accuracy of 60.06%, whereas the TwF model reached 54.02%. These results suggest that for physiological pain classification with limited data and high inter-subject variability, robust feature engineering combined with models possessing strong inductive biases is more effective than complex, end-to-end deep learning architectures.
Pain assessment is essential in clinical settings, but remains largely subjective, relying on verbal self-reports that are impractical for non-communicative patients. In this study, we propose an interpretable multimodal approach for automatic pain detection using physiological signals. Leveraging 65 participants (23 female, 42 male) multimodal data from the AI4Pain 2025 challenge, we systematically explore the contributions of multiple sensor modalities, Electrodermal Activity (EDA), Blood Volume Pulse (BVP), Respiration (RESP), and Peripheral Oxygen Saturation (SpO2) to classify pain into No Pain, Low Pain, and High Pain categories. After careful pre-processing and feature extraction, XGBoost outperformed other machine learning and deep learning models. In particular, the combination of only EDA and BVP features achieved 80% validation accuracy. In addition, the model using combined EDA and BVP features achieved high classification accuracy across all pain levels, particularly for High Pain (AUC = 0.92) and No Pain (AUC = 0.96), demonstrating strong discriminative ability. Using all modalities, the XGBoost model achieved a test accuracy of 60.92%, marking an 8.62% improvement over the multimodal test baseline. Notably, our model outperformed the baseline results across all modalities on both the validation and test sets. Our model also incorporates HRV features derived from BVP and EDA emphasizes interpretability through analysis of permutation feature importance. The results demonstrate that multimodal physiological data, when processed effectively, enable accurate and explainable pain detection, a step toward real-time, deployable pain monitoring systems in healthcare.
Pain is a complex sensory, cognitive, and emotional experience that plays a vital role in diagnosis, treatment, and patient monitoring. Pain detection is critical for effective care but remains challenging due to its subjective nature, especially in individuals unable to communicate verbally. Traditional visual and auditory cues, such as facial expressions or vocalizations, can vary significantly across cultures and contexts and may be consciously masked, limiting their reliability. Physiological signals, in contrast, offer objective and involuntary indicators of pain-related autonomic responses. We present PainXtract, a multimodal system submitted to the Second Multimodal Sensing Grand Challenge for Next-Gen Pain Assessment (AI4PAIN 2025). PainXtract classifies pain levels: no pain, low pain, and high pain, using handcrafted morphological features from electrodermal activity (EDA), blood volume pulse (BVP), respiratory signals (Resp), and peripheral oxygen saturation (SpO2). PainXtract achieves 75.57% accuracy on multiclass classification and near-perfect performance on binary classification, demonstrating the value of physiological signals and the central role of EDA in pain assessment. We also conduct extensive studies to assess the contribution of each modality. Our system outperforms the baseline system on both the validation and testing sets.
Pain assessment is a challenging task due to the subjective nature of pain and variability between subjects. Having an objective way to assess pain could result in better quality of life for patients. As pain can cause changes in physiological responses, these signals are well-suited for pain recognition systems. Considering this, in this work, we conduct an investigation into unimodal vs. multimodal pain recognition using physiological signals for the First Multimodal Sensing Grand Challenge for Next-Gen Pain Assessment (AI4Pain). We perform an exhaustive fusion of Electrodermal Activity (EDA), Blood Volume Pulse (BVP), Respiration (Resp), and Peripheral Oxygen (SpO2), comparing both early and late fusion-based approaches. Along with this, we also analyze two deep learning architectures, namely Convolutional Neural Network (CNN) and CNN-LSTM, detailing which modalities and architectures are most promising for pain detection. Our results for this challenge are encouraging, showing that late fusion with a CNN-based approach can accurately recognize no pain vs. low and high levels of pain, outperforming the AI4Pain baseline results on validation and test data.
Social media platforms use multimodal data (e.g., text, images, behavioral patterns) to infer user characteristics for algorithmic profiling. To comply with privacy regulations like the GDPR, companies provide transparency tools, which are often hard for users to interpret—especially individuals with cognitive impairments (CIs), whose specific needs remain underexplored. It is still unclear (1) what information users need and (2) how it should be effectively and accessibly represented. We investigate transparency needs across cognitive abilities, using Large Language Models (LLMs) to create more understandable representations of profiling. An exploratory study with 45 participants—30 without CIs and 15 with CIs—was conducted under three conditions. After 15 minutes of social media browsing, participants received either (1) a verbal explanation of profiling, (2) LLM-generated interest segments, or (3) LLM-generated user personas (in general or Easy-to-Read German for participants with CIs), followed by a semi-structured interview. Thematic analysis of transcripts revealed concerns about data sensitivity, perceived consequences, and the influence of cognitive abilities. Merely showing users collected or inferred data—regardless of format—may not meet user transparency needs. Our findings suggest transparency tools must go beyond data representation to explain inference mechanisms and potential outcomes, tailored to the sensitivities of different cognitive user groups.
Text-to-image diffusion models demonstrate strong capabilities in generating photorealistic content across diverse domains. However, they remain limited in synthesizing clinically relevant facial anomalies, such as cleft lip, due to the lack of domain-specific representations and adaptation strategies. In this work, we introduce a method for domain-specialized image generation by adapting a publicly available multimodal diffusion model to synthesize prompt-based, realistic facial images of both pre-operative and post-operative cleft lip conditioned on a small set of real images. We compute quantitative metrics to evaluate the realism, identity safety, and diversity of the generated images, including face identity recognition (FIR), Frechet inception distance (FID), and learned perceptual image patch similarity (LPIPS). In addition, two medical experts independently rated a subset of the generated samples for anatomical plausibility and visual realism. Results show that the adapted model avoids identity leakage, outperforms previous GAN-based approaches in distributional similarity, and achieves average human ratings of 4.85 for realism and 4.81 for anatomical plausibility on a 5-point Likert scale. Beyond qualitative generation, we demonstrate the clinical utility of the generated images by training a lip anomaly detection model on synthetic samples, achieving an accuracy of 79% on real clinical data. These findings establish a new paradigm for adapting generative models toward generating diverse, clinically meaningful imagery with high fidelity and domain specificity.
