Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning
arxiv(2024)
摘要
This paper proposes a novel framework for multi-label image recognition
without any training data, called data-free framework, which uses knowledge of
pre-trained Large Language Model (LLM) to learn prompts to adapt pretrained
Vision-Language Model (VLM) like CLIP to multilabel classification. Through
asking LLM by well-designed questions, we acquire comprehensive knowledge about
characteristics and contexts of objects, which provides valuable text
descriptions for learning prompts. Then we propose a hierarchical prompt
learning method by taking the multi-label dependency into consideration,
wherein a subset of category-specific prompt tokens are shared when the
corresponding objects exhibit similar attributes or are more likely to
co-occur. Benefiting from the remarkable alignment between visual and
linguistic semantics of CLIP, the hierarchical prompts learned from text
descriptions are applied to perform classification of images during inference.
Our framework presents a new way to explore the synergies between multiple
pre-trained models for novel category recognition. Extensive experiments on
three public datasets (MS-COCO, VOC2007, and NUS-WIDE) demonstrate that our
method achieves better results than the state-of-the-art methods, especially
outperforming the zero-shot multi-label recognition methods by 4.7
MS-COCO.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要