Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels
arxiv(2023)
摘要
Zero-shot text rankers powered by recent LLMs achieve remarkable ranking
performance by simply prompting. Existing prompts for pointwise LLM rankers
mostly ask the model to choose from binary relevance labels like "Yes" and
"No". However, the lack of intermediate relevance label options may cause the
LLM to provide noisy or biased answers for documents that are partially
relevant to the query. We propose to incorporate fine-grained relevance labels
into the prompt for LLM rankers, enabling them to better differentiate among
documents with different levels of relevance to the query and thus derive a
more accurate ranking. We study two variants of the prompt template, coupled
with different numbers of relevance levels. Our experiments on 8 BEIR data sets
show that adding fine-grained relevance labels significantly improves the
performance of LLM rankers.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要