Concept – An Evaluation Protocol on Conversational Recommender Systems with System-centric and User-centric Factors
CoRR(2024)
摘要
The conversational recommendation system (CRS) has been criticized regarding
its user experience in real-world scenarios, despite recent significant
progress achieved in academia. Existing evaluation protocols for CRS may
prioritize system-centric factors such as effectiveness and fluency in
conversation while neglecting user-centric aspects. Thus, we propose a new and
inclusive evaluation protocol, Concept, which integrates both system- and
user-centric factors. We conceptualise three key characteristics in
representing such factors and further divide them into six primary abilities.
To implement Concept, we adopt a LLM-based user simulator and evaluator with
scoring rubrics that are tailored for each primary ability. Our protocol,
Concept, serves a dual purpose. First, it provides an overview of the pros and
cons in current CRS models. Second, it pinpoints the problem of low usability
in the "omnipotent" ChatGPT and offers a comprehensive reference guide for
evaluating CRS, thereby setting the foundation for CRS improvement.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要