LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores.
CoRR(2023)
摘要
Automatic evaluation of generated textual content presents an ongoing
challenge within the field of NLP. Given the impressive capabilities of modern
language models (LMs) across diverse NLP tasks, there is a growing trend to
employ these models in creating innovative evaluation metrics for automated
assessment of generation tasks. This paper investigates a pivotal question: Do
language model-driven evaluation metrics inherently exhibit bias favoring texts
generated by the same underlying language model? Specifically, we assess
whether prominent LM-based evaluation metrics--namely, BARTScore, T5Score, and
GPTScore--demonstrate a favorable bias toward their respective underlying LMs
in the context of summarization tasks. Our findings unveil a latent bias,
particularly pronounced when such evaluation metrics are used in an
reference-free manner without leveraging gold summaries. These results
underscore that assessments provided by generative evaluation models can be
influenced by factors beyond the inherent text quality, highlighting the
necessity of developing more dependable evaluation protocols in the future.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要