Model Collapse Demystified: The Case of Regression
CoRR(2024)
摘要
In the era of large language models like ChatGPT, the phenomenon of "model
collapse" refers to the situation whereby as a model is trained recursively on
data generated from previous generations of itself over time, its performance
degrades until the model eventually becomes completely useless, i.e the model
collapses. In this work, we study this phenomenon in the simplified setting of
kernel regression and obtain results which show a clear crossover between where
the model can cope with fake data, and a regime where the model's performance
completely collapses. Under polynomial decaying spectral and source conditions,
we obtain modified scaling laws which exhibit new crossover phenomena from fast
to slow rates. We also propose a simple strategy based on adaptive
regularization to mitigate model collapse. Our theoretical results are
validated with experiments.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要