Emotion-Aware Multimodal Fusion for Meme Emotion Detection
IEEE Transactions on Affective Computing(2024)
摘要
The ever-evolving social media discourse has witnessed an overwhelming use of
memes to express opinions or dissent. Besides being misused for spreading
malcontent, they are mined by corporations and political parties to glean the
public's opinion. Therefore, memes predominantly offer affect-enriched insights
towards ascertaining the societal psyche. However, the current approaches are
yet to model the affective dimensions expressed in memes effectively. They rely
extensively on large multimodal datasets for pre-training and do not generalize
well due to constrained visual-linguistic grounding. In this paper, we
introduce MOOD (Meme emOtiOns Dataset), which embodies six basic emotions. We
then present ALFRED (emotion-Aware muLtimodal Fusion foR Emotion Detection), a
novel multimodal neural framework that (i) explicitly models emotion-enriched
visual cues, and (ii) employs an efficient cross-modal fusion via a gating
mechanism. Our investigation establishes ALFRED's superiority over existing
baselines by 4.94
best approaches on the challenging Memotion task. We then discuss ALFRED's
domain-agnostic generalizability by demonstrating its dominance on two
recently-released datasets - HarMeme and Dank Memes, over other baselines.
Further, we analyze ALFRED's interpretability using attention maps. Finally, we
highlight the inherent challenges posed by the complex interplay of disparate
modality-specific cues toward meme analysis.
更多查看译文
关键词
Emotion analysis,information fusion,memes,multimodality,social media
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要