Large Language Models excel in machine translation but face challenges with high computational costs and latency, making knowledge transfer to smaller models crucial. Traditional distillation methods, however, overlook student models’ existing capabilities, leading to redundant training on mastered knowledge and insufficient coverage of novel content. This paper presents WTA-MT, a Weakness-Targeted Augmentation framework for targeted knowledge transfer from LLMs to smaller models. By focusing on the student’s specific weaknesses instead of indiscriminate distillation, and leveraging LLM teachers to generate targeted supplementary data, WTA-MT efficiently enhances performance. Experiments in both general (En → Zh, De → En) and medical (En → De) domains demonstrate that WTA-MT outperforms traditional Sequence Knowledge Distillation (SeqKD) methods, achieving comparable or superior performance with significantly less data. Notably, our method eliminates performance deficiencies in low-scoring samples and while preserving high-quality translations, effectively narrowing the knowledge gap between small and large models.
Multi-domain machine translation remains a challenge when domain labels are unavailable during inference. Prior approaches typically assume a explicit domain from inputs, limiting their applicability in real-world scenarios. We propose a lightweight yet effective LLM-assisted framework that eliminates the requirement for explicit domain labels. Our method explores two paradigms: (1) LLM as Router, where a large language model predicts the domain and routes the input to the corresponding expert model; and (2) LLM as Ranker, where the LLM selects the best output among multiple expert candidates. Experiments on four diverse domains show that the our method achieves an average BLEU score of 43.53, only 1.62 BLEU lower than the upper-bound expert KNN-MT model with gold domain labels (45.15), while requiring no explicit domain labels. Moreover, while expert in-domain models suffer sharply deterioration on out-of-domain inputs, our method maintains strong robustness, effectively mitigating domain shift. The proposed methods combine the cross-domain adaptability of LLMs with the strong in-domain accuracy of expert NMT models, offering a plug-and-play, training-free solution without additional training.
Recent advances in Large Speech-Language Models (LSLMs) demonstrate strong speech understanding and cross-modal interaction abilities. However, the lack of standardized evaluation methods hinders their development. Existing evaluation approaches face three limitations:(1) Inconsistent datasets prevent fair model comparisons; (2) Current benchmarks focus on specific speech tasks but fail to assess responses to direct speech instructions; (3) Critical aspects like security and robustness are overlooked. To address these issues, we propose ISBench, a benchmark for evaluating LSLMs’ instruction-following capability and safety. Our framework introduces acoustic scenario simulations covering speaker characteristics (gender/age/emotion), environmental factors (background noise), and linguistic variations (colloquial expressions). Through comprehensive experiments with seven open-source models, we reveal key findings: LSLMs show performance gaps between speech and text modalities, exhibit weaker performance with children’s voices, and demonstrate significant sensitivity to noise and informal language. IS-Bench provides researchers with a unified evaluation platform to advance LSLM development.
Sentiment analysis in low-resource languages faces several challenges. This paper addresses the challenges of extremely low-resource and methodologically understudied sentiment analysis for Tigrigna, an official language spoken in Eritrea and the Tigray region of Ethiopia. We introduce TIGSEN, the first large-scale, multi-domain benchmark dataset for Tigrigna sentiment analysis, comprising 68,596 annotated text samples from official social media, news, and review forums. Created through a rigorous native-speakers annotation protocol, the dataset is designed to enable robust model training and evaluation. To prove its utility and to establish a strong, reproducible baseline for the community, we propose and evaluate a systematic cross-transfer learning framework. This methodology deliberately leverages annotated data from high-resource to linguistically related languages, English and Amharic, to overcome the limitations of Tigrigna’s small data pool. Our experiments show that models fine-tuned directly on TIGSEN achieved competitive performance. At the same time, the proposed cross-transfer framework yields a significant performance gain, achieving an accuracy of 87.6
While Large Language Models (LLMs) have shown promising capabilities in machine translation, their outputs often lack controllability and the ability to leverage error correction for translation improvement. To address this, we propose a two-stage framework, Diagnose-Then-Optimize (DTO), for structured translation quality enhancement. In the first stage, we fine-tune the large language model using human-annotated error data, enabling it to leverage translation error information for translation correction. In the second stage, we construct a preference dataset using response comparisons evaluated by ChatGPT, focusing on error correctness, correction effectiveness. We apply Direct Preference Optimization (DPO) to refine the model’s output behaviors based on these preferences. Our method demonstrates strong post-editing capabilities, consistently improving translation quality across WMT23 different systems’ outputs. The most significant gains are observed in English-Chinese, highlighting the model’s effectiveness in correcting diverse and complex translation errors. Experiments on WMT23 datasets across English–German, English–Russian, and English–Chinese demonstrate that DTO consistently improves the base LLaMA-3-8B, outperforming large-scale machine translation models such as NLLB_Greedy and Aya-23-35B in COMET scores. Our results highlight the effectiveness of combining structured error supervision with preference-driven fine-tuning, offering a robust and interpretable solution for controllable translation correction.
Utilizing a small amount of domain knowledge to achieve high-quality domain-specific translation is a challenging task. Nowadays, Large language model(LLM) is capable of generating more fluent and human-preferred translations through personalized instructions. However, in the field of domain-specific machine translation, the performance of LLM is inferior to traditional methods due to the lack of domain training data and the absence of domain transfer ability. To address the issue, we decided to incorporate terminology knowledge, which is crucial for accurately capturing the precise semantics of domain-specific texts. We design two types of terminology alignment instructions to enhance the model's cross-linguistic terminology alignment capability, explicitly integrating terminology knowledge into the model training process. According to the experiment, the model fine-tuned with MT+G-Align significantly outperformed the baseline through terminology translation accuracy and translation quality, demonstrating the effectiveness of the terminology alignment instructions. On the WMT 2023 Terminology Translation task, experimental results show that our approach achieves the best results in all three directions, including German-to-English, Chinese-to-English, and English-to-Chinese.
This paper mainly introduces the basic situation of Lan-Bridge's participation in the CCMT 2024 machine translation evaluation project. In this evaluation, we participate in the bilingual translation projects for three minority languages: Mongolian to Chinese, Tibetan to Chinese, and Uygur to Chinese. We adopt the Transformer model based on the self-attention network as our foundation and train three machine translation models for these three language pairs. The paper primarily discusses the specific methods and experimental details employed by the model, providing an in-depth analysis and discussion of the model performance in the bilingual translation tasks for the three minority languages.
Nearest-Neighbor Language Models (kNN-MT) leverage the contextual representations and next-word predictions of tokens to construct a vector-based database. During the inference stage, this database is utilized to assist the model in predicting the next word, resulting in impressive performance improvements. However, as the volume of data grows, the storage requirements for the vector-based database in kNN-MT continue to increase. Furthermore, the kNN retrieval performed for each predicted token introduces additional latency during the inference stage. To address these limitations, we propose training a lightweight neural network as a substitute for the vector datastore and kNN search. Our approach significantly reduces the storage overhead while maintaining fast inference speed, as demonstrated by our experiments on various translation datasets.
Joint modeling or multi-task learning by applying speech and text encoders can improve the performance of end-to-end speech translation (E2E ST) with the help of large-scale speech recognition and text translation data in previous work. However, most existing methods require architectural changes of ST training, and the modality gap between speech and text makes it hard to encode both of them with a shared encoder. In this paper, we regard the joint modeling for speech translation and text translation as multilingual NMT modeling, using a single encoder for both speech and text processing. We empirically show that the modality gap can be effectively addressed by modeling modality-aware relative position encoding in the self-attention layer. Experiments on three benchmarks covering 23 languages show that joint modeling over internal/external speech recognition and text translation data, the performance of our single encoder method can lead to significant improvements on multiple ST tasks in both directions (from and to English) compared to the baselines.
With the rapid development of large language models, a lot of preliminary work has been done to explore the performance of large language models on different artificial intelligence tasks. Despite this, research on evaluating the translation performance of large language models for Southeast Asian low-resource languages is still relatively scarce. The main reason is the scarcity of public evaluation data for Southeast Asian low-resource languages and large language models that can be directly injected into Southeast Asian low-resource languages for training. First, evaluation data for Southeast Asian low-resource language is extracted based on the publicly available Asian Language Treebank (ALT) corpus. Then, large language models with the capability to understand Southeast Asian low-resource languages are evaluated. Finally, three strategies are employed to construct in-context learning prompts to further enhance the translation performance of the existing large language models. Experimental results on multiple benchmark datasets demonstrate that large language models have superior contextual learning capabilities and their translation performances can be improved significantly by utilizing high-quality prompts. However, the guidance effect of sub-optimal prompts on the model is inconsistent. The comparative experiment further elucidated that the model temperature parameter exhibits distinct optimal values depending on the scale of text input. The subsequent analysis indicates that a higher degree of similarity between the prompt and the context facilitates the model's ability to generate accurate translation results.
This paper presents a human-in-the-loop approach to address the challenge of low-resource neural machine translation (NMT), focusing on the Tibetan-Chinese language pair. We emphasize the crucial role of human feedback in both data augmentation and model optimization. First, we construct a large-scale Tibetan-Chinese parallel corpus by iteratively leveraging back-translation and incorporating human evaluation to guide the generation of high-quality synthetic data. Then, we train a multilingual NMT system using a curriculum learning strategy, progressively incorporating the augmented data. Finally, we fine-tune our model with GaLore and SimPO algorithms, directly optimizing it towards human preferences as assessed by professional translators. Experimental results on the CCMT 2024 Tibetan-Chinese translation task demonstrate that our approach significantly improves translation quality, achieving state-of-the-art performance. We provide further analysis and case studies to illustrate the effectiveness of our human-in-the-loop methodology.
This paper presents the system architecture and technical details adopted by Institute of Scientific and Technical Information of China (ISTIC) during the evaluation at the 20th China Conference on Machine Translation (CCMT 2024). ISTIC participated in two evaluation tasks of machine translation: the English-Chinese News Domain Task and the Multi-Domain Machine Translation Task. The paper mainly discusses a translation method that combines the Transformer architecture with polishing by Qwen, data preprocessing methods.
Large language models (LLMs) have exhibited remarkable performance in various natural language processing tasks. In this paper, we describe our LLM-based machine translation system submitted to CCMT 2024 evaluation task. We investigated the effects of pre training data ratio, language quantity, instruction fine-tuning data ratio, and instruction construction method on the machine translation performance of large language models. We found that large language models have greater potential for multi-domain machine translation compared to traditional machine translation models. Within limits, increasing the number of newly added languages can enhance the overall translation capabilities for these new languages.
In this paper, we present our submission to the bilingual machine translation task of CCMT 2024 for the Chinese ↔ English constrained scenarios. Our systems are based on the Transformer architecture, the submitted system is an ensemble of multiple models of Transformer-variant for all directions. Only the released constrained data were used as our training data, with fine-grained data filtering and augmentation strategies. Additionally, we used various training techniques such as HyPe tuning, back-translation, forward translation, R-Drop, and alternated translation. The experimental results show that R-Drop, increasing data diversity, and model ensemble are the most effective methods in enhancing model performance.
This report presents the evaluation results of the Tibetan-Chinese machine translation task conducted by the National Language Resources Monitoring and Research Center for Minority Languages at the 20th Conference on Machine Translation (CCMT2024). The translation system described in this paper is based on the LLaMA2-7B architecture, with specific secondary pre-training and supervised fine-tuning tailored to the translation task. The results demonstrate that this approach effectively improves the performance of Tibetan-Chinese machine translation. Additionally, this paper employs a parameter grid search method to investigate the impact of different parameter combinations on translation performance.
Quality Estimation (QE) is the prediction of translation quality without reference to the translation. Critical Error Detection (CED), as a sub-task of the QE task, aims to detect and identify significant meaning biases in machine translation that may cause serious damage. State-of-the-art CED models are supervised: they need to be trained on data obtained by multiple professionals labeled with specific CED labels based on the output of some machine translation system, thus making them dependent on the content of the training set, the scarcity of which and the non-uniformity in the distribution of the labels can affect performance. In order to solve the above problems, in this paper, we propose a training-free CED approach-CED based on anchors test, our approach only needs to test the anchors in the source language by cross-lingual masking test, and then determine whether the translation contains critical errors based on the test results. We conducted experiments on blind test data from the WMT2021 CED sharing task using different models. The results show the effectiveness of our proposed method, reaching the performance of the supervised baseline on En-Zh, but overall there is still a gap compared to supervised CED, and we give an example to illustrate the potential of our method for explainable applications.
Generating adversarial examples contributes to mainstream neural machine translation (NMT) robustness. However, popular adversarial policies are apt for fixed tokenization, hindering its efficacy for common character perturbations involving versatile tokenization. Based on existing adversarial generation via reinforcement learning (RL), we propose the `DexChar policy' that introduces character perturbations for the existing mainstream adversarial policy based on token substitution. Furthermore, we improve the self-supervised matching that provides feedback in RL to cater to the semantic constraints required during training adversaries. Experiments show that our method is compatible with the scenario where baseline adversaries fail, and can generate high-efficiency adversarial examples for analysis and optimization of the system.
This paper presents the submission of Huawei Translation Services Center (HW-TSC) to machine translation tasks of the 20th China Conference on Machine Translation (CCMT 2024). We participate in the bilingual machine translation task and multi-domain machine translation task. For these two translation tasks, we use training strategies such as regularized dropout, bidirectional training, data diversification, forward translation, back translation, alternated training, curriculum learning, and transductive ensemble learning to train neural machine translation (NMT) models based on the deep Transformer-big architecture. Furthermore, to explore whether large language model (LLM) can effectively improve the translation quality of NMT models, we use supervised fine-tuning (SFT) to train llama2-13b as an Automatic post-editing (APE) model to improve the translation results of the NMT model on the multi-domain machine translation task. By using these plyometric strategies, our submission achieves a competitive result in the final evaluation.
Quality Estimation is a task aiming to estimate the quality of translations without relying on any references. This paper describes our submission for CCMT 2023 quality estimation sentence-level task for English-to-Chinese (EN-ZH). Due to the challenges of costly annotations and small dataset sizes in the QE field, many researchers have attempted to leverage rich parallel corpora for unsupervised learning through methods such as uncertainty quantification, and data augmentation. Existing mainstream unsupervised QE methods exhibit good diversity and variability, so we test these methods individually as well as their ensemble effect.
