Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination
CoRR(2024)
Abstract
The potential of Vision-Language Models (vlms) often remains
underutilized in handling complex text-based problems, particularly when these
problems could benefit from visual representation. Resonating with humans'
ability to solve complex text-based problems by (1) creating a visual diagram
from the problem and (2) deducing what steps they need to take to solve it, we
propose Self-Imagine. We leverage a single Vision-Language Model
(vlm) to generate a structured representation of the question using
HTML, then render the HTML as an image, and finally use the same to answer
the question using both the question and the image. Our approach does not
require any additional training data or training. We evaluate our approach in
three mathematics tasks and nine general-purpose reasoning tasks using
state-of-the-art vlm. Our approach boosts the performance of
vlm on all math tasks (: +4.62%; : +4.49%; :
+9.30%) and the majority of the general-purpose reasoning tasks by 0.4% to
13.20% while achieving comparable performance in other tasks.
Code and data at https://github.com/snat1505027/self-imagine .
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined