Revisiting pretraining for semi-supervised learning in the low-label regime

NEUROCOMPUTING(2024)

引用 0|浏览8
暂无评分
摘要
Semi -supervised learning (SSL) addresses the lack of labeled data by exploiting large unlabeled data through pseudolabeling. However, in the extremely low -label regime, pseudo labels could be incorrect, a.k.a. the confirmation bias, and the pseudo labels will in turn harm the network training. Recent studies combined finetuning (FT) from pretrained weights with SSL to mitigate the challenges and claimed superior results in the low -label regime. In this work, we first show that the better pretrained weights brought in by FT account for the state-of-the-art performance, and importantly that they are universally helpful to off -the -shelf semi -supervised learners. We further argue that direct finetuning from pretrained weights is suboptimal due to covariate shift and propose a contrastive target pretraining step to adapt model weights towards target dataset. We carried out extensive experiments on both classification and segmentation tasks by doing target pretraining then followed by semi -supervised finetuning. The promising results validate the efficacy of target pretraining for SSL, in particular in the low -label regime.
更多
查看译文
关键词
Semi-supervised learning,Transfer learning,Self-supervised learning,Contrastive learning
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要