FeatUp: A Model-Agnostic Framework for Features at Any Resolution
ICLR 2024(2024)
摘要
Deep features are a cornerstone of computer vision research, capturing image
semantics and enabling the community to solve downstream tasks even in the
zero- or few-shot regime. However, these features often lack the spatial
resolution to directly perform dense prediction tasks like segmentation and
depth prediction because models aggressively pool information over large areas.
In this work, we introduce FeatUp, a task- and model-agnostic framework to
restore lost spatial information in deep features. We introduce two variants of
FeatUp: one that guides features with high-resolution signal in a single
forward pass, and one that fits an implicit model to a single image to
reconstruct features at any resolution. Both approaches use a multi-view
consistency loss with deep analogies to NeRFs. Our features retain their
original semantics and can be swapped into existing applications to yield
resolution and performance gains even without re-training. We show that FeatUp
significantly outperforms other feature upsampling and image super-resolution
approaches in class activation map generation, transfer learning for
segmentation and depth prediction, and end-to-end training for semantic
segmentation.
更多查看译文
关键词
deep learning,deep features,computer vision,feature upsampling
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要