Generative Information Retrieval Evaluation
arxiv(2024)
摘要
In this chapter, we consider generative information retrieval evaluation from
two distinct but interrelated perspectives. First, large language models (LLMs)
themselves are rapidly becoming tools for evaluation, with current research
indicating that LLMs may be superior to crowdsource workers and other paid
assessors on basic relevance judgement tasks. We review past and ongoing
related research, including speculation on the future of shared task
initiatives, such as TREC, and a discussion on the continuing need for human
assessments. Second, we consider the evaluation of emerging LLM-based
generative information retrieval (GenIR) systems, including retrieval augmented
generation (RAG) systems. We consider approaches that focus both on the
end-to-end evaluation of GenIR systems and on the evaluation of a retrieval
component as an element in a RAG system. Going forward, we expect the
evaluation of GenIR systems to be at least partially based on LLM-based
assessment, creating an apparent circularity, with a system seemingly
evaluating its own output. We resolve this apparent circularity in two ways: 1)
by viewing LLM-based assessment as a form of "slow search", where a slower IR
system is used for evaluation and training of a faster production IR system;
and 2) by recognizing a continuing need to ground evaluation in human
assessment, even if the characteristics of that human assessment must change.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要