VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context
arxiv(2024)
摘要
Large Multimodal Models (LMMs) have achieved impressive success in visual
understanding and reasoning, remarkably improving the performance of
mathematical reasoning in a visual context. Yet, a challenging type of visual
math lies in the multimodal graph theory problem, which demands that LMMs
understand the graphical structures accurately and perform multi-step reasoning
on the visual graph. Additionally, exploring multimodal graph theory problems
will lead to more effective strategies in fields like biology, transportation,
and robotics planning. To step forward in this direction, we are the first to
design a benchmark named VisionGraph, used to explore the capabilities of
advanced LMMs in solving multimodal graph theory problems. It encompasses eight
complex graph problem tasks, from connectivity to shortest path problems.
Subsequently, we present a Description-Program-Reasoning (DPR) chain to enhance
the logical accuracy of reasoning processes through graphical structure
description generation and algorithm-aware multi-step reasoning. Our extensive
study shows that 1) GPT-4V outperforms Gemini Pro in multi-step graph
reasoning; 2) All LMMs exhibit inferior perception accuracy for graphical
structures, whether in zero/few-shot settings or with supervised fine-tuning
(SFT), which further affects problem-solving performance; 3) DPR significantly
improves the multi-step graph reasoning capabilities of LMMs and the GPT-4V
(DPR) agent achieves SOTA performance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要