GM2RC: Graph-based Multitask Modality Refinement and Complement for Multimodal Sentiment Analysis

Multimodal sentiment analysis has emerged as a critical research area, aiming to analyze complex emotional states using data from multiple sources. While conventional approaches focus on sophisticated fusion techniques to integrate multimodal information, they often struggle with distributional variances among modalities. Bridging this modal distribution gap and optimizing the utilization of modality-sensitive information are crucial challenges. In response, our study presents GM2RC, an innovative framework designed to enhance modality representations for improved multimodal fusion efficacy. GM2RC adopts a two-step approach: intra-modal refinement and intermodal complementation. In the first step, it enhances distinct features within each modality, while the second step fosters the assimilation of shared information across modalities through pairwise learning, thus mitigating inter-modal disparity. This comprehensive approach leads to more accurate sentiment analysis by creating more robust multimodal representations. Further rigorous experiments demonstrate significant boosts of 1.59% in accuracy and 2.03% in w-F1 on the IEMOCAP dataset, and 1.65% in accuracy and 2.73% in w-F1 on the MELD dataset, showcasing the superiority of our proposed GM2RC model.