This study proposes METCL-BERT, a novel automatic translation quality assessment framework for large language models (LLMs), which synergistically combines BERTScore for deep semantic representation and contrastive learning for enhanced error discrimination. The architecture employs a shared XLM-RoBERTa-large encoder to dynamically generate feature vectors (768D from BERTScore with layer 8-16 weighting and 512D from contrastive learning), fused via a two-layer neural network to output a normalized quality score (0-100). Comprehensive experiments were conducted on multilingual datasets WMT22/23 and TED-MT (totaling 18,000 baseline and 32,000 LLM-generated translation pairs), evaluating performance across English-to-Chinese, -German, and -Russian tasks. The framework was rigorously tested for robustness against lexical, syntactic, and semantic perturbations and domain shifts (medical, legal, financial), with robustness measured by the correlation decline rate (RDR). Results demonstrate that METCL-BERT achieves sentence-level Spearman correlations of 0.791 (en-zh), 0.803 (en-de), and 0.782 (en-ru), significantly outperforming the best baseline KIWI-22 by >7.6%. It attains a system-level Kendall Tau of 0.832, markedly superior to COMET-22 (0.745). Crucially, its robustness is validated by an average RDR of 18.70% across perturbation tests, substantially lower than BERTScore (24.50%) and COMET-22 (21.20%). Further strengths include exceptional discriminative power (QSD=2.14) with strictly increasing quality interval medians (92.5→78.0→65.0→38.0) and a large effect size (Cohen's d=4.37). Ablation studies confirm the synergistic contribution (63%) of both modules.
Autosomal recessive polycystic kidney disease (ARPKD) is caused primarily by mutations in PKHD1, encoding fibrocystin (FPC), but Pkhd1 mutant mice failed to reproduce the human phenotype. In contrast, the renal lesion in c ongenital p olycystic k idney ( cpk ) mice, with a mutation in Cys1 and cystin protein loss, closely phenocopies ARPKD. Although the nonhomologous mutation diminished the translational relevance of the cpk model, recent identification of patients with CYS1 mutations and ARPKD prompted the investigations described herein. We examined cystin and FPC expression in mouse models ( cpk , rescued‐ cpk (r‐ cpk ), Pkhd1 mutants) and mouse cortical collecting duct (CCD) cell lines (wild type ( wt ), cpk ). We found that cystin deficiency caused FPC loss in both cpk kidneys and CCD cells. FPC levels increased in r ‐cpk kidneys and siRNA of Cys1 in wt cells reduced FPC. However, FPC deficiency in Pkhd1 mutants did not affect cystin levels. Cystin deficiency and associated FPC loss impacted the architecture of the primary cilium, but not ciliogenesis. No reduction in Pkhd1 mRNA levels in cpk kidneys and CCD cells suggested posttranslational FPC loss. Studies of cellular protein degradation systems suggested selective autophagy as a mechanism. In support of the previously described function of FPC in E3 ubiquitin ligase complexes, we demonstrated reduced polyubiquitination and elevated levels of functional epithelial sodium channel in cpk cells. Therefore, our studies expand the function of cystin in mice to include inhibition of Myc expression via interaction with necdin and maintenance of FPC as functional component of the NEDD4 E3 ligase complexes. Loss of FPC from E3 ligases may alter the cellular proteome, contributing to cystogenesis through multiple, yet to be defined, mechanisms.
<p>Supplementary Table 1 shows clinicopathological characteristics of the patients studies in this project.</p>
<p>This file includes supplementary methods that describe the details of transwell assay, RT-PCR, western blotting, immunohistochemistry and CTC enrichment by NanoVelcro system.</p>
Autosomal recessive polycystic kidney disease (ARPKD) is caused primarily by mutations in PKHD1, encoding fibrocystin (FPC), but Pkhd1 mutant mice failed to reproduce the human phenotype. In contrast, the renal lesion in congenital polycystic kidney (cpk) mice, with a mutation in Cys1 and cystin protein loss, closely phenocopies ARPKD. Although the nonhomologous mutation diminished the translational relevance of the cpk model, recent identification of patients with CYS1 mutations and ARPKD prompted the investigations described herein. We examined cystin and FPC expression in mouse models (cpk, rescued-cpk (r-cpk), Pkhd1 mutants) and mouse cortical collecting duct (CCD) cell lines (wild type (wt), cpk). We found that cystin deficiency caused FPC loss in both cpk kidneys and CCD cells. FPC levels increased in r-cpk kidneys and siRNA of Cys1 in wt cells reduced FPC. However, FPC deficiency in Pkhd1 mutants did not affect cystin levels. Cystin deficiency and associated FPC loss impacted the architecture of the primary cilium, but not ciliogenesis. No reduction in Pkhd1 mRNA levels in cpk kidneys and CCD cells suggested posttranslational FPC loss. Studies of cellular protein degradation systems suggested selective autophagy as a mechanism. In support of the previously described function of FPC in E3 ubiquitin ligase complexes, we demonstrated reduced polyubiquitination and elevated levels of functional epithelial sodium channel in cpk cells. Therefore, our studies expand the function of cystin in mice to include inhibition of Myc expression via interaction with necdin and maintenance of FPC as functional component of the NEDD4 E3 ligase complexes. Loss of FPC from E3 ligases may alter the cellular proteome, contributing to cystogenesis through multiple, yet to be defined, mechanisms.
This article shows that the structure of bilingual material from standard parsing and alignment tools is not optimal for training syntax-based statistical machine translation (SMT) systems. We present three modifications to the MT training data to improve the accuracy of a state-of-the-art syntax MT system: re-structuring changes the syntactic structure of training parse trees to enable reuse of substructures; re-labeling alters bracket labels to enrich rule application context; and re-aligning unifies word alignment across sentences to remove bad word alignments and refine good ones. Better structures, labels, and word alignments are learned by the EM algorithm. We show that each individual technique leads to improvement as measured by BLEU, and we also show that the greatest improvement is achieved by combining them. We report an overall 1.48 BLEU improvement on the NIST08 evaluation set over a strong baseline in Chinese/English translation.
We use the Margin Infused Relaxed Algorithm of Crammer et al. to add a large number of new features to two machine translation systems: the Hiero hierarchical phrase-based translation system and our syntax-based translation system. On a large-scale Chinese-English translation task, we obtain statistically significant improvements of +1.5 Bleu and + 1.1 Bleu, respectively. We analyze the impact of the new features and the performance of the learning algorithm.
We use the Margin Infused Relaxed Algorithm of Crammer et al. to add a large number of new features to two machine translation systems: the Hiero hierarchical phrasebased translation system and our syntax-based translation system. On a large-scale ChineseEnglish translation task, we obtain statistically significant improvements of +1.5 B and +1.1 B, respectively. We analyze the impact of the new features and the performance of the learning algorithm.
We compare and contrast the strengths and weaknesses of a syntax-based machine translation model with a phrase-based machine translation model on several levels. We briefly describe each model, highlighting points where they differ. We include a quantitative comparison of the phrase pairs that each model has to work with, as well as the reasons why some phrase pairs are not learned by the syntax-based model. We then evaluate proposed improvements to the syntax-based extraction techniques in light of phrase pairs captured. We also compare the translation accuracy for all variations.
We show that phrase structures in Penn Treebank style parses are not optimal for syntaxbased machine translation. We exploit a series of binarization methods to restructure the Penn Treebank style trees such that syntactified phrases smaller than Penn Treebank constituents can be acquired and exploited in translation. We find that by employing the EM algorithm for determining the binarization of a parse tree among a set of alternative binarizations gives us the best translation result.
Designers of statistical machine translation (SMT) systems have begun to employ treestructured translation models. Systems involving tree-structured translation models tend to be complex. This article aims to reduce the conceptual complexity of such systems, in order to make them easier to design, implement, debug, use, study, understand, explain, modify, and improve. In service of this goal, the article extends the theory of semiring parsing to arrive at a novel abstract parsing algorithm with five functional parameters: a logic, a grammar, a semiring, a search strategy, and a termination condition. The article then shows that all the common algorithms that revolve around tree-structured translation models, including hierarchical alignment, inference for parameter estimation, translation, and structured evaluation, can be derived by generalizing two of these parameters — the grammar and the logic. The article culminates with a recipe for using such generalized parsers to train, apply, and evaluate an SMT system that is driven by tree-structured translation models.
We present a probabilistic bilingual capitalization model for capitalizing machine translation outputs using conditional random fields. Experiments carried out on three language pairs and a variety of experiment conditions show that our model significantly outperforms a strong monolingual capitalization model baseline, especially when working with small datasets and/or European language pairs.
Statistical MT has made great progress in the last few years, but current translation models are weak on re-ordering and target language fluency. Syntactic approaches seek to remedy these problems. In this paper, we take the framework for acquiring multi-level syntactic translation rules of (Galley et al., 2004) from aligned tree-string pairs, and present two main extensions of their approach: first, instead of merely computing a single derivation that minimally explains a sentence pair, we construct a large number of derivations that include contextually richer rules, and account for multiple interpretations of unaligned words. Second, we propose probability estimates and a training procedure for weighting these rules. We contrast different approaches on real examples, show that our estimates based on multiple derivations favor phrasal re-orderings that are linguistically better motivated, and establish that our larger rules provide a 3.63 BLEU point increase over minimal rules.
We introduce SPMT, a new class of statistical Translation Models that use Syntactified target language Phrases. The SPMT models outperform a state of the art phrase-based baseline model by 2.64 Bleu points on the NIST 2003 Chinese-English test corpus and 0.28 points on a human-based quality metric that ranks translations on a scale from 1 to 5.
MANOS (Multilingual Application Network for Olympic Services) project. aims to provide intelligent multilingual information services in 2008 Olympic Games. By narrowing down the general language technology, this paper gives an overview of our new work on Phrase-Based Statistical Machine Translation (PBT) under the framework of the MANOS. Starting with the construction of large scale Chinese-English corpus (sentence aligned) and introduction four methods to extract phrases, The promising results from PBT systems lead us to confidences for constructing a high-quality translation system and harmoniously integrate it into MANOS platform.