Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
CoRR(2024)
Abstract
Instruction tuning is critical to large language models (LLMs) for achieving
better instruction following and task adaptation capabilities but its success
heavily relies on the training data quality. Many recent methods focus on
improving the data quality but often overlook the compatibility of the data
with the student model being finetuned. This paper introduces Selective
Reflection-Tuning, a novel paradigm that synergizes a teacher LLM's reflection
and introspection for improving existing data quality with the data selection
capability of the student LLM, to automatically refine existing
instruction-tuning data. This teacher-student collaboration produces
high-quality and student-compatible instruction-response pairs, resulting in
sample-efficient instruction tuning and LLMs of superior performance. Selective
Reflection-Tuning is a data augmentation and synthesis that generally improves
LLM finetuning and self-improvement without collecting brand-new data. We apply
our method to Alpaca and WizardLM data and achieve much stronger and top-tier
7B and 13B LLMs. Our codes, models, and data will be released at
https://github.com/tianyi-lab/Reflection_Tuning.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined