Self-Refinement of Language Models from External Proxy Metrics Feedback
CoRR(2024)
摘要
It is often desirable for Large Language Models (LLMs) to capture multiple
objectives when providing a response. In document-grounded response generation,
for example, agent responses are expected to be relevant to a user's query
while also being grounded in a given document. In this paper, we introduce
Proxy Metric-based Self-Refinement (ProMiSe), which enables an LLM to refine
its own initial response along key dimensions of quality guided by external
metrics feedback, yielding an overall better final response. ProMiSe leverages
feedback on response quality through principle-specific proxy metrics, and
iteratively refines its response one principle at a time. We apply ProMiSe to
open source language models Flan-T5-XXL and Llama-2-13B-Chat, to evaluate its
performance on document-grounded question answering datasets, MultiDoc2Dial and
QuAC, demonstrating that self-refinement improves response quality. We further
show that fine-tuning Llama-2-13B-Chat on the synthetic dialogue data generated
by ProMiSe yields significant performance improvements over the zero-shot
baseline as well as a supervised fine-tuned model on human annotated data.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要