CaMML: Context-Aware Multimodal Learner for Large Models
CoRR(2024)
摘要
In this work, we introduce Context-Aware MultiModal Learner (CaMML), for
tuning large multimodal models (LMMs). CaMML, a lightweight module, is crafted
to seamlessly integrate multimodal contextual samples into large models,
thereby empowering the model to derive knowledge from analogous,
domain-specific, up-to-date information and make grounded inferences.
Importantly, CaMML is highly scalable and can efficiently handle lengthy
multimodal context examples owing to its hierarchical design. Based on CaMML,
we have developed two multimodal models, CaMML-7B and CaMML-13B, that have
shown exceptional performance across an array of benchmark datasets for
multimodal tasks. Remarkably, CaMML-13B achieves the state-of-the-art
performance on over ten widely recognized multimodal benchmark datasets,
surpassing LLaVA-1.5 (13B) with a noticeable margin, without integration of any
external resources. Moreover, we have conducted extensive ablative studies to
inspect the inner workings of CaMML and performed qualitative analyses to
showcase its effectiveness in handling real-world challenging cases.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要