LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
CoRR(2024)
摘要
Inspired by the outstanding zero-shot capability of vision language models
(VLMs) in image classification tasks, open-vocabulary object detection has
attracted increasing interest by distilling the broad VLM knowledge into
detector training. However, most existing open-vocabulary detectors learn by
aligning region embeddings with categorical labels (e.g., bicycle) only,
disregarding the capability of VLMs on aligning visual embeddings with
fine-grained text description of object parts (e.g., pedals and bells). This
paper presents DVDet, a Descriptor-Enhanced Open Vocabulary Detector that
introduces conditional context prompts and hierarchical textual descriptors
that enable precise region-text alignment as well as open-vocabulary detection
training in general. Specifically, the conditional context prompt transforms
regional embeddings into image-like representations that can be directly
integrated into general open vocabulary detection training. In addition, we
introduce large language models as an interactive and implicit knowledge
repository which enables iterative mining and refining visually oriented
textual descriptors for precise region-text alignment. Extensive experiments
over multiple large-scale benchmarks show that DVDet outperforms the
state-of-the-art consistently by large margins.
更多查看译文
关键词
Open Vocabulary Object Detection,Visual descriptors
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要