Understanding Multi-Structured Documents Via LLMs’

Complexly structured data present in documents and web content pose significant challenges for accurate MLLM reasoning. Although MLLMs have advanced substantially, they continue to struggle with intricate data formats such as nested tables and multi-dimensional charts, often leading to hallucinations. This paper explores the capabilities of LLMs and MLLMs in understanding and answering questions from complex data found in PDF documents by leveraging a pre-processing pipeline consisting of industrial and open-source tools. Our results showcase that incorporating RAG and pre-processing tools enables MLLMs to achieve approximately 5 https://github.com/manulife-ai/financialqa .