Domain-specific Multimodal Neural Machine Translation (DMNMT) aims to translate text in specialized domains by leveraging both linguistic context and associated visual information to resolve domain-specific ambiguities and enhance terminological accuracy. Although accompanying images often provide sparse and fragmented visual cues that could potentially anchor critical domain semantics, the semantic mapping from images to textual domain semantics typically exhibits sparse multi-focal alignment challenges. Existing general-domain multimodal neural machine translation (MNMT) models and large language models (LLMs) struggle to achieve accurate aggregation of domain-salient information, often resulting in near-equivalent yet imprecise terminology translations or outright errors. To bridge this sparse domain semantic correspondence gap, we introduce the Asymmetric Siamese Multimodal Fusion (ASMF) framework, which decouples domain representation learning into two complementary branches that both consume text: a domain-specific virtual visual content generation (DVVG) branch and a terminology-aware textual (TAT) branch. The DVVG branch distills sparse, localized visual features into modality-agnostic semantic anchors through mask-constrained multi-focal distillation, while the TAT branch captures terminology-dense textual context. We introduce a novel Domain-Virtualized Pivot-driven Hierarchical Fusion (DVPH) strategy that progressively injects distilled visual anchors across encoder layers. This asymmetric dual-branch design effectively couples spatially fragmented visual details with terminology-rich text, enabling accurate and domain-consistent translations even for low-frequency terms. Extensive experiments were conducted on four benchmark datasets covering three distinct scenarios: two domain-specific datasets (Fashion-MMT and EMMT), one general-domain dataset (Multi30K), and one multi-domain dataset (WIT). Comprehensive evaluations demonstrate that the proposed approach outperforms existing MNMT, DMNMT and LLMs, achieving state-of-the-art (SOTA) results across all datasets. In-depth analyses validate its robustness and generalization capabilities across diverse scenarios, including visually noisy or image-free conditions.

Rapid and accurate disaster classification is critical for effective emergency response and resource allocation. Existing approaches to multimodal crisis classification typically presume that the textual and visual elements of a tweet share the same label, thereby overlooking frequent mismatches between modalities and neglecting the contextual links that span multiple tweets. To address these limitations, we introduce GNN-MCC, a novel Graph Neural Network based framework for Multimodal Crisis Classification. Our method constructs a unified graph representation in which individual tweets are decomposed into separate text and image nodes, allowing the model to capture complex intra- and inter-modal relationships. To further reconcile discrepancies between modalities, we incorporate a contrastive attention mechanism that aligns complementary information across text and image channels. Empirical evaluation on the publicly available CrisisMMD dataset demonstrates that GNN-MCC substantially outperforms state-of-the-art baselines under both label-consistent and label-inconsistent conditions. The source code for this paper is available at https://github.com/kliean/GNN-MCC .

Knowledge distillation is an effective technique for compressing neural machine translation models. However, traditional word-level knowledge distillation only transfers word-level probabilities. This leads to a problem: the student model’s encoder cannot effectively model the syntactic structure of the source language. Consequently, this may cause structural disarray in the translated output. Integrating prior syntactic information and selectively learning valuable knowledge can significantly enhance neural machine translation performance.To address this, we propose a selective self-attention knowledge distillation method with encoder-side dependency constraints. The method transforms a source-language dependency syntax tree into an adjacency matrix. This matrix is then applied to the weighted computation of the self-attention network. Furthermore, the method introduces a selective distillation strategy. This strategy identifies and transfers knowledge from the teacher model that possesses greater training utility. Additionally, explicit syntactic constraints are embedded into the self-attention mechanisms of both the teacher and student models. Structural alignment distillation is employed to acquire cross-lingual alignment knowledge. This integrated approach enhances the distillation effect.Experimental results on multiple translation tasks demonstrate the effectiveness of our method. The tasks include WMT’14 En-De, IWSLT’14 En-De/En-Fr/En-Ar, and IWSLT’15 En-Vi. Our method effectively improves translation quality. For instance, on WMT’14 En-De, our approach achieves improvements of +1.57 BLEU and +8.19 COMET over the Transformer-Base model, with particularly significant gains on long sentences. The joint application of dependency-constrained self-attention and selective knowledge distillation effectively enhances machine translation performance.

Domain-specific Multimodal Neural Machine Translation (DMNMT) aims to integrate image information associated with textual data to enhance translation performance for domain-specific texts. However, due to the low frequency and sparse distribution of domain-specific terminology, the model tends to favor learning general knowledge. This bias leads to insufficient modeling of domain-specific features, limiting the accuracy and reliability of translating domain terms. To address this, we propose a two-stage incremental semantic aggregation framework through an incremental training mechanism. In the first stage, the model learns general semantic representations and foundational translation capabilities. In the second stage, a contrastive decoupling strategy and a multi-level collaborative decoding mechanism are introduced to strengthen the model’s perception and modeling of domain-specific multimodal features. Subsequently, a dual-stage balancing optimization strategy is employed to harmonize translation performance between domain-specific terminology and general vocabulary, thereby improving overall translation quality. Experimental results demonstrate that our model outperforms state-of-the-art (SOTA) methods on multiple benchmark datasets, achieving significant BLEU score improvements of 2.29 and 2.13 on domain-specific and general datasets, respectively. Further ablation studies and visualization analyses confirm the model’s effectiveness in translating domain terms and its outstanding robustness. Our code is available at https://github.com/lyy-nlp/DMNMT_Two-stage.

Domain-specific Multimodal Machine Translation (DMNMT) faces a critical robustness challenge due to the diverse and unpredictable visual noise prevalent in user-uploaded images. Although incremental learning can adapt to different noise types step by step, it cannot exhaustively cover all possible noise distributions and incurs high cost and complexity, causing existing models to struggle to maintain stable performance, particularly under unseen noise. To address this challenge, this paper presents a non-incremental recursive mutual-supervised multi-modal variational fusion approach, which anchors domain semantics in text and learns modality-agnostic representations. Specifically, we treat paired image-text data as mutually self-supervised information and apply an adaptive mutual-supervised mask-variational distillation strategy to generate modality-agnostic representations, using text as the domain backbone. Then, we introduce a reverse visual-to-textual domain-anchor alignment strategy to promote domain-adaptive machine translation, ensuring that modality-agnostic information is effectively integrated into the text and enhancing generalization across diverse visual noise conditions. We conduct extensive experiments on three benchmark datasets, covering domain-specific and general domains. The results demonstrate that our method achieves state-of-the-art (SOTA) performance on all test sets. Moreover, our approach exhibits robust stability across nine different noise scenarios, particularly in real-world unseen noisy environments.

Domain-specific multimodal neural machine translation (DMNMT) aims to translate source language domain sentences into target language by incorporating images as additional contextual information. However, domain-specific multimodal scenarios frequently suffer from visual imbalance issues, such as one sentence corresponding to multiple images or even no images at all. Effectively integrating visual information into text to enhance domain sentence translation performance under visual imbalance issues is one of the critical challenges for DMNMT, especially for domain-related terms. To tackle these domain-specific visual imbalance problems, this article introduces a virtual domain distillation-enhanced multimodal fusion with the awareness of multiview correlations to enhance the robustness and performance of domain machine translation across various multimodal domain scenarios. We first adopt a multiview correlation-aware cross-modal distillation strategy to generate virtual domain visual scenes by extracting visual correlations among all images through multikernel representations. Subsequently, we integrate pseudo-domain visual scenes into text to improve the performance of domain-specific machine translation. Our proposed approach has the ability to capture domain visual representations across different scenarios, and contributing to more effective domain-specific translation. We conduct expensive experiments on three domain-specific and general-domain benchmark datasets. Experimental results demonstrate that our proposed approach achieves state-of-the-art (SOTA) machine translation scores on most test sets. The in-depth analysis demonstrates the effectiveness and robustness of our proposed approach for domain machine translation.

Domain-specific multimodal neural machine translation (DMNMT) aims to incorporate text-related visual details into text to enhance machine translation performance in specific domains. Existing DMNMT approaches often borrow cross-modal fusion strategies from multi-modal neural machine translation (MNMT). However, these DMNMT methods rely heavily on their corresponding visual scene information, resulting in a lack of robustness in translating domain-specific expressions. This issue, known as the “domain semantic shift problem,” is particularly evident in noisy scene scenarios and even text-only scenarios. To this end, this paper adopts an Uncertain Aspect-aware Triplet Contrastive Transferring strategy (UAs-TriCT) through a cross-modal regularized variational disentangling-distillation mechanism for DMNMT. Our proposed approach enhances the model's ability to generate semantically aligned target sentences while preserving sensitivity to domain-specific aspects. Specifically, we propose a regularized variational disentangling-distillation strategy to capture modality- and linguistic-shared Uncertain Aspect-aware (UAs) representations. This mechanism can generate modality-shared virtual visual representations from the text through two types of regularized distillation, even in noisy or text-only scenarios. Then, we employ an UAs-TriCT strategy to ensure cross-modal and cross-linguistic semantic alignment while maintaining UAs-center awareness during target sentence generation. We conduct experiments on four benchmark MNMT datasets, covering domain-specific, multi-domain, and general-domain scenarios. The experimental results demonstrate that our proposed approach achieves state-of-the-art performance across all test sets for both MNMT and DMNMT datasets. The in-depth analysis further validates the effectiveness of the disentangling-distillation and UAs-TriCT strategies across aligned-image, noisy-image, or even text-only scenarios.

Multimodal emotion recognition in conversation (MERC) has attracted increasing attention for its wide applications. While previous methods have focused on extracting and merging features from various modalities, they often do not sufficiently consider interactions within and across modalities, as well as the semantic and modal correlations of utterances within a dialogue. To address this gap, we propose a dual-level constraint-based distributed GCN model. This model constructs a graph convolution network by simultaneously incorporating multiple modal-central graphs to capture the multimodal features of utterances. Additionally, we adopt a dual-level constraint strategy to identify the semantic and modal correlation of each speaker’s utterances in a dialogue. By comprehensively capturing the utterance features, our model significantly enhances emotion classification performances. The comparative experiment results show that our model outperforms the state-of-the-art models on the benchmark datasets IEMOCAP, MELD and M3ED. In addition, we verify the effectiveness of each component of our model by ablation experiments.

Lexical analysis is a fundamental task in natural language processing, which involves several subtasks, such as word segmentation (WS), part-of-speech (POS) tagging, and named entity recognition (NER). Recent works have shown that taking advantage of relatedness between these subtasks can be beneficial. This paper proposes a unified neural framework to address these subtasks simultaneously. Apart from the sequence tagging paradigm, the proposed method tackles the multitask lexical analysis via two-stage sequence span classification. Firstly, the model detects the word and named entity boundaries by multi-label classification over character spans in a sentence. Then, the authors assign POS labels and entity labels for words and named entities by multi-class classification, respectively. Furthermore, a Gated Task Transformation (GTT) is proposed to encourage the model to share valuable features between tasks. The performance of the proposed model was evaluated on Chinese and Thai public datasets, demonstrating state-of-the-art results.

Multimodal aspect-based sentiment analysis (MABSA) identifies the sentiment polarity of specific aspects in text with the aid of accompanying images. Current methods often exhibit limited robustness to visual noise and scene adaptability, particularly under visually noisy conditions or image absence, leading to suboptimal performance. To address this, we propose a Prompt-based Dual-layer Cross-modal Distillation (PDCD) learning strategy. PDCD leverages aspect term prompts to focus the model on relevant aspect semantics and employs a dual-layer cross-modal distillation mechanism integrating textual and visual features. This approach progressively extracts valuable visual cues while suppressing noise, enabling accurate aspect sentiment prediction even with missing images or mismatched image-text pairs. The proposed PDCD offers two key innovations: (1) its dual-layer cross-modal distillation framework demonstrates strong adaptability to diverse scenarios and noise types. (2) Its prompt-guided dual-aspect representation effectively aggregates aspect semantics within noisy visual conditions. Experiments on Twitter2015, Twitter2017, and MASAD benchmarks show PDCD outperforms most existing state-of-the-art (SOTA) methods by 75.69 F1-score on average. Additional visual noise tests confirm superior robustness across diverse noise scenarios, achieving the highest F1-scores. In-depth analyses further validate the strategy’s effectiveness in noisy MABSA scenarios.

Multimodal Sentiment Analysis (MSA) aims to integrate multiple information sources, such as text and visual modalities, to predict the sentiments expressed in the data. However, the scarcity of multimodal datasets poses a significant challenge for effective vision-language fusion. Prompt learning has recently emerged as a promising solution to this issue. Existing prompting methods either convert image regions into visual tokens that align with textual word dimensions using a visual encoder or directly use image features. However, these methods overlook the inherent differences between the characteristics of a visual encoder and a language model, making it challenging for the language model to directly comprehend the semantic information of the images. To address this, we propose an adaptive multimodal prompt-tuning approach for sentiment analysis. Specifically, in the proposed adaptive multimodal prompt generation module, we extract initial multimodal features using an advanced pre-trained model, ensuring comprehensive integration of information from different modalities. We then introduce and fuse a learnable vector with the initial multimodal features dynamically, creating contextually relevant multimodal prompts. Finally, we integrate these multimodal prompts with the maked text sequence vector and send them to a pre-trained language model to obtain the word probability distribution. This process enhances the language model’s ability to comprehend and utilize semantic information from multimodal inputs, especially images. Extensive experiments and analyses on two aspect-level and two sentence-level datasets demonstrate that our method outperforms existing state-of-the-art approaches, confirming the effectiveness of the proposed adaptive multimodal prompts.

With the rapid popularity of short video content, multimodal sentiment analysis (MSA) has attracted extensive attention. Most previous MSA studies have focused on manually transcribed benchmark datasets, which are both costly to generate and limited in availability. In real-world applications, MSA often relies on Automatic Speech Recognition (ASR) technology. However, due to the noise in the captions generated by ASR, traditional MSA models suffer a substantial decline in performance. To address this problem, this paper proposes a Multi-level Sentiment-aware Clustering Denoising Model (MSCDM), which effectively enhances the robustness by introducing sentiment distance constraints both intra- and inter-modality. Specifically, the model first compensates for the loss of sentiment semantic information in the text modality caused by ASR by leveraging samples with the same sentiment polarity to guide each other. Subsequently, the model refines cross-modal sentiment representations by dividing samples with multimodal information into positive and negative examples. We conduct extensive experiments on real-world datasets including MOSI-SpeechBrain, MOSI-IBM, and MOSI-iFlytek, and the results demonstrate the model’s effectiveness, outperforming the current state-of-the-art models on three datasets. The in-depth analysis confirms the performance of the multi-level clustering denoising strategy proposed for MSA.

Graph Representation Learning (GRL) can be fundamentally modeled as a physical process of seeking an energy equilibrium state for a node system on a latent manifold. However, existing Graph Neural Networks (GNNs) often suffer from uncontrolled energy dissipation during message passing, driving the system towards a state of Thermal Death–manifested as feature collapse or over-smoothing–due to the absence of explicit thermodynamic constraints. To address this, we propose HyperGRL, a thermodynamics-driven framework that embeds nodes on a unit hypersphere by minimizing a Helmholtz free energy objective composed of two competing potentials. First, we introduce Structural Binding Energy (via Neighbor-Mean Alignment), which functions as a local binding force to strengthen structural cohesion, encouraging structurally related nodes to form compact local clusters. Second, to counteract representation collapse, we impose a Mean-Field Repulsive Potential (via Sampling-Free Uniformity), which acts as a global entropic force to maximize representation dispersion without the need for negative sampling. Crucially, to govern the trade-off between local alignment and global uniformity, we devise an Adaptive Thermostat. This entropy-guided strategy dynamically regulates the system's "temperature" during training, guiding the representation towards a robust metastable state that balances local cohesion with global discriminability. Extensive experiments on node classification, node clustering, and link prediction show that HyperGRL consistently achieves strong performance across diverse benchmark datasets, yielding more discriminative and robust representations while alleviating over-smoothing.

Multi-modal Machine Translation (MMT) aims to tackle the challenge of cross-modal semantic alignment by integrating additional information from additional modalities, such as images, video, audio, and potentially other modalities. Unfortunately, collecting high-quality multi-modal data pairs is costly, leading to challenges in data scarcity or noise robustness. Most existing MMT research focuses on feature-level cross-modal fusion using these limited multi-modal data, training models from scratch without utilizing prior knowledge from established pure-text neural machine translation (NMT) models. This results in inefficient use of computational resources and cross-modal misalignment. To this end, this paper presents a triplet-modality group-guided incremental distillation approach, constrained by group-centered multi-modal semantic alignment, to extend the scope of machine translation in visual scenarios. The proposed approach preserves the translation capabilities of the pre-trained NMT model through triplet-modal group incremental distillation, while further improving translation performance through a regularized group alignment strategy, thereby enhancing machine translation ability in MMT. We conducted extensive experiments on two general-domain and two specific-domain MMT tasks. The results demonstrate that the proposed approach shows improvements over the state-of-the-art (SOTA) methods across all test sets, achieving performance gains of over 3.7%. In-depth analysis highlights the effectiveness and robustness of our method in cross-modal alignment and noisy visual scenarios. Our code is available at https://github.com/lyy-nlp/MMT_main.

Pre-trained language models, such as Bidirectional Encoder Representations from Transformers (BERT), have demonstrated state-of-the-art performance in many Natural Language Processing (NLP) downstream tasks. Incorporating pre-trained BERT knowledge into the Sequence-to-Sequence (Seq2Seq) model can significantly enhance machine translation performance, particularly for low-resource language pairs. However, most previous studies prefer to fine-tune both the large pre-trained BERT model and the Seq2Seq model jointly, leading to costly training times, especially with limited parallel data pairs. Consequently, the integration of pre-trained BERT contextual representations into the Seq2Seq framework is limited. In this paper, we propose a simple and effective BERT knowledge fusion approach based on regularized Mixup for low-resource Neural Machine Translation (NMT), referred to as ReMixup-NMT, which constrains the distributions of the normal Transformer encoder and the Mixup-based Transformer encoder to be consistent. The proposed ReMixup NMT approach is able to distill and fuse the pre-trained BERT knowledge into Seq2Seq NMT architecture in an efficient manner with non-additional parameters training. Experiment results on six low-resource NMT tasks show the proposed approach outperforms the state-of-the-art (SOTA) BERT-fused and drop-based methods on IWSLT'15 English -> Vietnamese and IWSLT'17 English -> French datasets.

Multi-modal Aspect-Based Sentiment Analysis (MABSA) aims to identify the sentiment polarity of aspects by incorporating visual information into text. Image and text are two types of modality information with significant modality gaps in both data form and semantic expression. Narrowing the modality gaps and feature fusion are two crucial challenges in MABSA. To address these issues, this paper introduces an aspect-enhanced alignment and fusion strategy with dual-layer contrastive learning to tackle the cross-modal fusion problem. Unlike traditional contrastive learning methods, our approach increases the number of negative samples, enabling the model to learn more discriminative features and better capture fine-grained cross-modal relationships. The proposed approach leverages overlapping aspect information as multi-modal pivots to first bridge the modality gaps and then integrate visual and text information in the multi-modal feature space, thereby improving multi-modal sentiment analysis performance. We first introduce an aspect-guided modality alignment strategy that narrows the fundamental modality gaps between image and text using modality contrastive learning. Then, we design an aspect-oriented multi-modal fusion approach to promote cross-modal feature fusion through symmetric cross-modal interaction. Extensive experiments demonstrate that the proposed approach outperforms other state-of-the-art (SOTA) MABSA methods on three MABSA benchmark datasets. In-depth analysis further validates the effectiveness of the proposed multi-modal fusion approach for MABSA.

Syllabification is a crucial task in natural language processing, and syllables also play a significant role as modeling units in speech processing. While deep learning methods have shown remarkable progress in syllabification, they face challenges in low-resource languages where ready-made segmentation datasets or rules are lacking. Large language models (LLMs) are mostly unsuitable for these low-resource languages as well. To address these challenges, this paper proposes an unsupervised syllabification approach that incorporates logical reasoning into the reinforcement learning training process, achieving knowledge-guided syllabification. By introducing logical reasoning knowledge and modeling the interaction between the Agent and the knowledge base (KB), the model gains a better understanding of language structures and patterns. The study primarily focuses on low-resource Lao language, with experiments conducted on a publicly available English dataset to validate the effectiveness of the proposed method.

Domain-specific Multi-modal Neural Machine Translation (DMNMT) aims to translate domain-specific sentences from a source language to a target language by incorporating text-related visual information. Generally, domain-specific text-image data often complement each other and have the potential to collaboratively enhance the representation of domain-specific information. Unfortunately, there is a considerable modality gap between image and text in data format and semantic expression, which leads to distinctive challenges in domain-text translation tasks. Narrowing the modality gap and improving domain-aware representation are two critical challenges in DMNMT. To this end, this paper proposes a progressive modality-complement aggregative MultiTransformer, which aims to simultaneously narrow the modality gap and capture domain-specific multi-modal representation. We first adopt a bidirectional progressive cross-modal interactive strategy to effectively narrow the text-to-text, text-to-visual, and visual-to-text semantics in the multi-modal representation space by integrating visual and text information layer-by-layer. Subsequently, we introduce a modality-complement MultiTransformer based on progressive cross-modal interaction to extract the domain-related multi-modal representation, thereby enhancing machine translation performance. Experiment results on the Fashion-MMT and Multi-30k datasets are conducted, and the results show that the proposed approach outperforms the compared state-of-the-art (SOTA) methods on the En-Zh task in E-commerce domain, En-De, En-Fr and En-Cs tasks of Multi-30k in general domain. The in-depth analysis confirms the validity of the proposed modality-complement MultiTransformer and bidirectional progressive cross-modal interactive strategy for DMNMT.

Document-level neural machine translation (DNMT) faces challenges in integrating context and maintaining semantic consistency. This paper proposes a coarse-to-fine semantic transfer approach that captures both fine-grained and coarse-grained document-level context. By employing a two-stage training strategy with Adapter-enhanced modules, the method ensures effective semantic adaptation from sentence to document levels. Experiments on three datasets demonstrate significant improvements in s-BLEU and d-BLEU metrics, outperforming existing approaches.

The goal of multi-modal neural machine translation (MNMT) is to incorporate language-agnostic visual information into text to enhance the performance of machine translation. However, due to the inherent differences between image and text, these two modalities inevitably suffer from semantic mismatch problems. To tackle this issue, this paper adopts a multi-grained visual pivot-guided multi-modal fusion strategy with cross-modal contrastive disentangling to eliminate the linguistic gaps between different languages. By using the disentangled multi-grained visual information as a cross-lingual pivot, we can enhance the alignment between different languages and improve the performance of MNMT. We first introduce text-guided stacked cross-modal disentangling modules to progressively disentangle image into two types of visual information: MT-related visual and background information. Then we effectively integrate these two kinds of multi-grained visual elements to assist target sentence generation. Extensive experiments on four benchmark MNMT datasets are conducted, and the results demonstrate that our proposed approach achieves significant improvement over the other state-of-the-art (SOTA) approaches on all test sets. The in-depth analysis highlights the benefits of text-guided cross-modal disentangling and visual pivot-based multi-modal fusion strategies in MNMT. We release the code at https://github.com/nlp-mnmt/ConVisPiv-MNMT.