REASONS: A benchmark for REtrieval and Automated citationS Of scieNtific Sentences using Public and Proprietary LLMs
arxiv(2024)
摘要
Automatic citation generation for sentences in a document or report is
paramount for intelligence analysts, cybersecurity, news agencies, and
education personnel. In this research, we investigate whether large language
models (LLMs) are capable of generating references based on two forms of
sentence queries: (a) Direct Queries, LLMs are asked to provide author names of
the given research article, and (b) Indirect Queries, LLMs are asked to provide
the title of a mentioned article when given a sentence from a different
article. To demonstrate where LLM stands in this task, we introduce a large
dataset called REASONS comprising abstracts of the 12 most popular domains of
scientific research on arXiv. From around 20K research articles, we make the
following deductions on public and proprietary LLMs: (a) State-of-the-art,
often called anthropomorphic GPT-4 and GPT-3.5, suffers from high pass
percentage (PP) to minimize the hallucination rate (HR). When tested with
Perplexity.ai (7B), they unexpectedly made more errors; (b) Augmenting relevant
metadata lowered the PP and gave the lowest HR; (c) Advance retrieval-augmented
generation (RAG) using Mistral demonstrates consistent and robust citation
support on indirect queries and matched performance to GPT-3.5 and GPT-4. The
HR across all domains and models decreased by an average of 41.93
was reduced to 0
Score and BLEU were 68.09
adversarial samples showed that LLMs, including the Advance RAG Mistral,
struggle to understand context, but the extent of this issue was small in
Mistral and GPT-4-Preview. Our study con tributes valuable insights into the
reliability of RAG for automated citation generation tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要