CoUDA: Coherence Evaluation via Unified Data Augmentation
arxiv(2024)
摘要
Coherence evaluation aims to assess the organization and structure of a
discourse, which remains challenging even in the era of large language models.
Due to the scarcity of annotated data, data augmentation is commonly used for
training coherence evaluation models. However, previous augmentations for this
task primarily rely on heuristic rules, lacking designing criteria as guidance.
In this paper, we take inspiration from linguistic theory of discourse
structure, and propose a data augmentation framework named CoUDA. CoUDA breaks
down discourse coherence into global and local aspects, and designs
augmentation strategies for both aspects, respectively. Especially for local
coherence, we propose a novel generative strategy for constructing augmentation
samples, which involves post-pretraining a generative model and applying two
controlling mechanisms to control the difficulty of generated samples. During
inference, CoUDA also jointly evaluates both global and local aspects to
comprehensively assess the overall coherence of a discourse. Extensive
experiments in coherence evaluation show that, with only 233M parameters, CoUDA
achieves state-of-the-art performance in both pointwise scoring and pairwise
ranking tasks, even surpassing recent GPT-3.5 and GPT-4 based metrics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要