Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model
arxiv(2022)
摘要
Explainable question answering systems should produce not only accurate
answers but also rationales that justify their reasoning and allow humans to
check their work. But what sorts of rationales are useful and how can we train
systems to produce them? We propose a new style of rationale for open-book
question answering, called markup-and-mask, which combines aspects of
extractive and free-text explanations. In the markup phase, the passage is
augmented with free-text markup that enables each sentence to stand on its own
outside the discourse context. In the masking phase, a sub-span of the
marked-up passage is selected. To train a system to produce markup-and-mask
rationales without annotations, we leverage in-context learning. Specifically,
we generate silver annotated data by sending a series of prompts to a frozen
pretrained language model, which acts as a teacher. We then fine-tune a smaller
student model by training on the subset of rationales that led to correct
answers. The student is "honest" in the sense that it is a pipeline: the
rationale acts as a bottleneck between the passage and the answer, while the
"untrusted" teacher operates under no such constraints. Thus, we offer a new
way to build trustworthy pipeline systems from a combination of end-task
annotations and frozen pretrained language models.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要