AmbigDocs: Reasoning across Documents on Different Entities under the Same Name
arxiv(2024)
摘要
Different entities with the same name can be difficult to distinguish.
Handling confusing entity mentions is a crucial skill for language models
(LMs). For example, given the question "Where was Michael Jordan educated?" and
a set of documents discussing different people named Michael Jordan, can LMs
distinguish entity mentions to generate a cohesive answer to the question? To
test this ability, we introduce a new benchmark, AmbigDocs. By leveraging
Wikipedia's disambiguation pages, we identify a set of documents, belonging to
different entities who share an ambiguous name. From these documents, we
generate questions containing an ambiguous name and their corresponding sets of
answers. Our analysis reveals that current state-of-the-art models often yield
ambiguous answers or incorrectly merge information belonging to different
entities. We establish an ontology categorizing four types of incomplete
answers and automatic evaluation metrics to identify such categories. We lay
the foundation for future work on reasoning across multiple documents with
ambiguous entities.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要