Assessing and Verifying Task Utility in LLM-Powered Applications
arxiv(2024)
摘要
The rapid development of Large Language Models (LLMs) has led to a surge in
applications that facilitate collaboration among multiple agents, assisting
humans in their daily tasks. However, a significant gap remains in assessing to
what extent LLM-powered applications genuinely enhance user experience and task
execution efficiency. This highlights the need to verify utility of LLM-powered
applications, particularly by ensuring alignment between the application's
functionality and end-user needs. We introduce AgentEval, a novel framework
designed to simplify the utility verification process by automatically
proposing a set of criteria tailored to the unique purpose of any given
application. This allows for a comprehensive assessment, quantifying the
utility of an application against the suggested criteria. We present a
comprehensive analysis of the effectiveness and robustness of AgentEval for two
open source datasets including Math Problem solving and ALFWorld House-hold
related tasks. For reproducibility purposes, we make the data, code and all the
logs publicly available at https://bit.ly/3w3yKcS .
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要