Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models
CoRR(2024)
摘要
While large language models (LMs) demonstrate remarkable performance, they
encounter challenges in providing accurate responses when queried for
information beyond their pre-trained memorization. Although augmenting them
with relevant external information can mitigate these issues, failure to
consider the necessity of retrieval may adversely affect overall performance.
Previous research has primarily focused on examining how entities influence
retrieval models and knowledge recall in LMs, leaving other aspects relatively
unexplored. In this work, our goal is to offer a more detailed, fact-centric
analysis by exploring the effects of combinations of entities and relations. To
facilitate this, we construct a new question answering (QA) dataset called
WiTQA (Wikipedia Triple Question Answers). This dataset includes questions
about entities and relations of various popularity levels, each accompanied by
a supporting passage. Our extensive experiments with diverse LMs and retrievers
reveal when retrieval does not consistently enhance LMs from the viewpoints of
fact-centric popularity.Confirming earlier findings, we observe that larger LMs
excel in recalling popular facts. However, they notably encounter difficulty
with infrequent entity-relation pairs compared to retrievers. Interestingly,
they can effectively retain popular relations of less common entities. We
demonstrate the efficacy of our finer-grained metric and insights through an
adaptive retrieval system that selectively employs retrieval and recall based
on the frequencies of entities and relations in the question.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要