Probing Multimodal Large Language Models for Global and Local Semantic Representations
arxiv(2024)
摘要
The advancement of Multimodal Large Language Models (MLLMs) has greatly
accelerated the development of applications in understanding integrated texts
and images. Recent works leverage image-caption datasets to train MLLMs,
achieving state-of-the-art performance on image-to-text tasks. However, there
are few studies exploring which layers of MLLMs make the most effort to the
global image information, which plays vital roles in multimodal comprehension
and generation. In this study, we find that the intermediate layers of models
can encode more global semantic information, whose representation vectors
perform better on visual-language entailment tasks, rather than the topmost
layers. We further probe models regarding local semantic representations
through object recognition tasks. We find that the topmost layers may
excessively focus on local information, leading to a diminished ability to
encode global information. Our code and data are released via
https://github.com/kobayashikanna01/probing_MLLM_rep.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要