MetaCoCo: A New Few-Shot Classification Benchmark with Spurious Correlation
arxiv(2024)
摘要
Out-of-distribution (OOD) problems in few-shot classification (FSC) occur
when novel classes sampled from testing distributions differ from base classes
drawn from training distributions, which considerably degrades the performance
of deep learning models deployed in real-world applications. Recent studies
suggest that the OOD problems in FSC mainly including: (a) cross-domain
few-shot classification (CD-FSC) and (b) spurious-correlation few-shot
classification (SC-FSC). Specifically, CD-FSC occurs when a classifier learns
transferring knowledge from base classes drawn from seen training distributions
but recognizes novel classes sampled from unseen testing distributions. In
contrast, SC-FSC arises when a classifier relies on non-causal features (or
contexts) that happen to be correlated with the labels (or concepts) in base
classes but such relationships no longer hold during the model deployment.
Despite CD-FSC has been extensively studied, SC-FSC remains understudied due to
lack of the corresponding evaluation benchmarks. To this end, we present Meta
Concept Context (MetaCoCo), a benchmark with spurious-correlation shifts
collected from real-world scenarios. Moreover, to quantify the extent of
spurious-correlation shifts of the presented MetaCoCo, we further propose a
metric by using CLIP as a pre-trained vision-language model. Extensive
experiments on the proposed benchmark are performed to evaluate the
state-of-the-art methods in FSC, cross-domain shifts, and self-supervised
learning. The experimental results show that the performance of the existing
methods degrades significantly in the presence of spurious-correlation shifts.
We open-source all codes of our benchmark and hope that the proposed MetaCoCo
can facilitate future research on spurious-correlation shifts problems in FSC.
The code is available at: https://github.com/remiMZ/MetaCoCo-ICLR24.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要